Is it possible for markdown export to link to other files?

This post was last updated for 204 days ago, and the information may already be changed

I exported a notebook to markdown (which is great), but I found that the files don't have links to other files like they were in Siyuan.

So, is there anyway to make markdown exports have links to other files?

    Welcome to here!

    Here we can learn from each other how to use SiYuan, give feedback and suggestions, and build SiYuan together.

    Signup About
    Please input reply content ...
    • 88250

      Sorry, converting references to relative paths to Markdown files is not supported.

      Some of the existing reference export modes can be seen in Settings - Export:

      image.png

    • sagar 2 1 Up
      VIP Warrior

      @88250 I think this is an important missing feature when it comes to publishing the notes. Let me explain:

      Many users, including me, like to make a lot of links between notes. We also like to publish some set of notes on our websites/blogs/digital gardens. With most other note-taking software, you can export the notes to markdown, which includes links to other markdown documents (or even specific paragraphs in the markdown documents). Then, static site generators like Hugo can ingest the markdown files and produce HTML pages for the web. These HTML pages have all the correct links to other notes' HTML pages. With SiYuan, since the exported markdown does not contain links to other notes, publishing notes with links to other notes is not possible at all. I think this is a problem for many potential users. Do you have any other suggestions for how to publish notes in SiYuan with all links-between-notes working properly?

      I think this is important for another reason: With other tools like Obsidian, the notes are stored as normal markdown files. This means that the notes are not "trapped" inside the note-taking software. SiYuan does not store content as markdown files. Even if the format is still open, it still feels like the notes are "trapped" or "locked inside" SiYuan. If SiYuan has an excellent ability to export the notes to markdown (e.g. with all the links preserved) it would feel less like the notes are locked away inside SiYuan. I think this will increase trust in SiYuan, since users will not worry that if SiYuan unfortunately disappears, then their notes will disappear with it.

      I understand that because SiYuan offers a lot more complex formatting options, exporting to Markdown with everything properly preserved is not possible. This comes from the limitations of Markdown. However, there should ideally be a way to publish the notes (perhaps to HTML) so that all the formatting and links are preserved. I think maybe this is currently possible for each entry in the Doc Tree, but ideally, it should be possible for an entire hierarchy in the Doc Tree i.e. a given note and all notes below it should be exported. Please let me know if this is already possible.

      1 Reply
      1 Operate
      sagar updated this reply at 2023-11-29 02:41:05
    • 88250

      Hello, you can try setting the reference export to footnote mode. We think this is the best ref export solution because most Markdown parsers support footnotes.

      1 Reply
    • adham
      Author

      I agree with @sagar that this is a very important feature which may increase SiYuan's popularity, especially against Obsidian.

    • sagar 2
      VIP Warrior

      I think there is a fairly simple solution to this, which I will describe below. I think it can be implemented as a simple Python script that does some post-processing of the exported markdown files. I will try to create a proof-of-concept script, but it will take me a few days because I am not a programmer and will need to learn some Python programming first.

      Solution description

      1. In Settings > Export configure 'Ref' to 'Anchor text with block URL'. After doing this, a link to another block in a SiYuan note appears in the exported Markdown file as [Anchor text](siyuan://blocks/block-id).

      2. Write a Python script that opens each exported markdown file and uses regex matching to identify all instances of [Anchor text](siyuan://blocks/block-id). For each instance,

      3. Use the /api/query/sql API endpoint to get details of the block-id by making a query like:

        { "stmt": "SELECT * FROM blocks WHERE id='block-id'" }

      4. In the query result, you can find the hpath key whose value is a human readable path.

        1. If the block-id is for a block of type document (which would be the case for a link to another document), then the Python script just needs to replace siyuan://blocks/block-id with hpath_value.md which would be the relative path to the exported markdown file that the link points to.
        2. If the block-id is for a block inside a document (e.g. to some arbitrary paragraph block in the document), all you need to do is to open the exported markdown document containing that paragraph (from the value of hpath for that block-id), find the start of the paragraph and add an <a name="block-id"></a> anchor at the start of the paragraph. Then, the siyuan://blocks/block-id in the exported markdown file can be replaced by hpath_value.md#block-id

      This would basically solve the problem of having correct links in the exported markdown files and additionally the exported notes can easily be converted to HTML and published e.g. with a static site generator like Hugo.

      @88250 Do you see any problems with this approach? I don't understand why SiYuan does not do this already when exporting to markdown.. but in any case, a Python script that does the above can post-process the exported markdown files from SiYuan and fix all the links. What do you think?

      1 Reply
      1 Operate
      sagar updated this reply at 2023-12-01 08:25:07
    • 88250

      Maybe it is feasible. If you have time, you can simply verify it and see if there are developers in the community to implement and improve it. Thank you.

    • adham
      Author

      I actually thought about doing that yesterday, and woke up today and opened my laptop to write that solution, only to find @sagar had written it. Funny!:)

    • adham
      Author

      I'll try it out.

    • adham 1
      Author

      I was able to do it (for documents only, not blocks, as I don't know how to find out if the link is of a block or a document, and I don't want to mutate the linked documents anyway).

      Here's the Python script:

      import os
      import re
      import requests
      
      # Replace 'YOUR_API_ENDPOINT' with the actual Siyuan API endpoint
      SIYUAN_API_ENDPOINT = "http://localhost:6806"
      
      
      # Function to make a SQL query to Siyuan API
      def query_block_details(block_id):
          query = {"stmt": f"SELECT * FROM blocks WHERE id='{block_id}'"}
          response = requests.post(f"{SIYUAN_API_ENDPOINT}/api/query/sql", json=query)
          return response.json()
      
      
      # Function to process each markdown file
      def process_markdown_file(file_path):
          with open(file_path, "r", encoding="utf-8") as file:
              content = file.read()
      
              # Use regex to find instances of [Anchor text](siyuan://blocks/block-id)
              matches = re.findall(r"\[.*?\]\(siyuan://blocks/(.*?)\)", content)
      
              for block_id in matches:
                  # Query Siyuan API to get block details
                  result = query_block_details(block_id)
      
                  # Assumes ref is of a document, not a block
                  if result:
                      hpath_value = result["data"][0].get("hpath")
      
                      # Replace the siyuan://blocks/block-id with hpath_value.md
                      new_link = f"({hpath_value}.md)"
      
                      # Replace the old link in the content
                      content = content.replace(f"(siyuan://blocks/{block_id})", new_link)
      
          # Write the updated content back to the file
          with open(file_path, "w", encoding="utf-8") as file:
              file.write(content)
      
      
      def process_markdown_directory(directory_path):
          for root, _, files in os.walk(directory_path):
              for file_name in files:
                  if file_name.endswith(".md"):
                      file_path = os.path.join(root, file_name)
                      process_markdown_file(file_path)
      
      
      # Replace 'YOUR_MARKDOWN_DIRECTORY_PATH' with the actual path to your exported markdown files
      markdown_directory_path = "YOUR_MARKDOWN_DIRECTORY_PATH"
      
      # Process each markdown file in the specified path
      process_markdown_directory(markdown_directory_path)
      
      
    • adham
      Author

      Thanks @sagar for your helpful solution!

    • adham
      Author

      Here's the before and after comparison.

      Before:

      
      # Networking
      
      The most important things to know are the networking protocols.
      
      
      * [TCP](siyuan://blocks/20231110002225-jzlfmvr)
      * [UDP](siyuan://blocks/20231111114912-9gi1l0p)
      * [HTTP](siyuan://blocks/20231110002250-b9w413n)
      * [TLS](siyuan://blocks/20231111114933-oti4rz4)
      * [WebSockets](siyuan://blocks/20231111115839-xw23n4k)
      * [WebRTC](siyuan://blocks/20231111114916-nevtkga)
      
      
      

      After:

      # Networking
      
      The most important things to know are the networking protocols.
      
      
      * [TCP](/Networking/TCP.md)
      * [UDP](/Networking/UDP.md)
      * [HTTP](/Networking/HTTP.md)
      * [TLS](/Networking/TLS.md)
      * [WebSockets](/Networking/WebSockets.md)
      * [WebRTC](/Networking/WebRTC.md)
      
      
      
    • sagar
      VIP Warrior

      @adham Wow 😍 You are a master programmer indeed! I was still trying to figure out how to use Python to open all .md files in a directory, by copy/pasting code from Stackoverflow 😄 This solution is exactly what I meant.

      I do believe it can be extended to include references to any arbitrary block, not just a document block. You did say that you don't want to mutate the linked documents, but I am just laying out the approach in case any other master programmers want to try it 😄. You wrote,

      I don't know how to find out if the link is of a block or a document,

      I see a few different possible checks for this

      • Look at the 'type' field in the response of the SQL query. If it is a document block, the type value will be 'd'. Other type values are given in the documentation at /Please Start Here/Advanced search/Type filtering
      • I believe that if the block is of type document, then the 'root_id' will be the same as 'id'. See the documentation of the SQL table fields at /Please Start Here/Advanced search/Database table

      So one or both of the above checks can be used to determine if the block is of type 'document' or something else.

      If the block is of type something else, then the way to link to it would be

      1. Open the target document that contains the block (i.e. the hpath)
      2. Search for the occurrence of the content in the 'markdown' field of the SQL query response. This will place the "cursor" inside the document to the start of the paragraph (or whatever other content) the block points to
      3. At this cursor position, insert an HTML anchor text like: <a name="block-id"></a>
      4. Then, update the siyuan://blocks/block-id part of the link to hpath_value.md#block-id

      This approach enables having links in the markdown files to any other markdown files or parts inside markdown files.

      @adham Thank you once again for your example code 🙏

      3 Operate
      sagar updated this reply at 2023-12-02 01:22:47
      sagar updated this reply at 2023-12-02 01:21:03
      sagar updated this reply at 2023-12-02 01:18:06
    • adham 1
      Author

      You're welcome! I actually was too lazy to write it myself so I asked ChatGPT to do it for me and changed some mistakes in its code.

      And I might as well do it for non-document blocks, I will post the solution here too.

      In any case, it seems like this problem can be solved easily and be integrated into SiYuan.
      I am not familiar with SiYuan code or Golang in general so someone else more familiar may make a Pull Request for this feature way faster than me.
      If not, I don't mind spending some time to learn enough to build this feature into SiYuan. It's the least I can do in return for getting this amazing app for free!
      Thanks @88250 @sagar and long live the open source!

    • giovannipollo

      Thanks a lot for the solution. It works really. I actively write python code for my PhD. If I have time, I will for sure look into extending it with blocks.

    • 88250 1

      @participants We are considering built-in support for anchor hash export mode Issue #10265 · siyuan-note/siyuan

      1 Reply
    • zxhd86

      Thanks to the solution provided by the community, the effect of the built-in Siyuan export is perfect! @adham @sagar

    • Siysuy

      The anchor hash feature currently creates a span object <span id="20231110120736-ze6m41f"></span>. Some kind of program should be able to understand and find this block inside the note as reference. Can anyone help me understand how this can be useful after export, where do I use this, which program understands this syntax?

      The export mode Anchor hash for notebook Markdown exporting has some limitations.

      In the case i want to publish a particular note (with links to other notes) my approach is:

      1. From main notebook export the notebook you want to publish to .sy.zip file
      2. Create new (temporary) notebook
      3. import .sy.zip file to new temp notebook
      4. Export temp notebook export it to any selectable format (only markdown is currently selectable)

      Other limitation of Anchor has export and the only way I got it to work (performed testing by exporting to Obsidian):

      • currently only works for notebook and does not allow nested notes. In the process of exporting all notes must be unnested inside notebook for .md export to function.
      • Currently does not support block reference. Can only link to main note. Tested by importing to obsidian (not a siyuan issue, but a markdown syntax limitation i presume)

      Depending on the responses here I might add a feature request soon that involves the concept of user selectable export syntax. In which before execution of export using anchor hash a user can type in the conversion syntax to apply. User is presented with input fields much like the settings > Export > Anchor text wrapping symbol fields.

      Conversion syntax for siyuan > obsidian would be: [[test3.md/Note 2#^d8bf62|Note 2]] and conversion input fields could somehow be [[ .. / .. #^ .. | .. ]]

      1 Reply
    • 88250

      In theory, programs that support standard Markdown (CommonMark) should be able to support parsing and rendering inline HTML elements, that is, <span>, but they may not necessarily support anchor hash jumps. After rendering from Markdown to HTML, the browser can support anchor hash jumping, so currently, exporting references as anchor hashs is the optimal solution.

      1 Reply
    • Siysuy

      I think the issue remains with anchor hash for the reason that it is limited to notebook currently, meaning I cannot select rendering from Markdown to HTML from the notebook_ex > export > (selectable=Markdown/siyuan.sy.zip).

      How would i export my notes to HTML with anchor hash? I think I would be limited to using pandoc input.md -o output.html, I tried it with pandoc, which succesfully creates html with link to other note, but it just opens the other note without navigation to the <span> block, so no hash jump in browser (brave/firefox) to block.

      1 Reply
      4 Operate
      Siysuy updated this reply at 2024-03-21 00:58:07
      Siysuy updated this reply at 2024-03-21 00:55:13
      Siysuy updated this reply at 2024-03-21 00:41:56
      Siysuy updated this reply at 2024-03-21 00:34:11
    • 88250

      The reason why this function is placed at the notebook level is because some users no longer want to use SiYuan and need complete batch export, so support for this function is not considered at the document level.

      I'm not sure if the span tag still exists after pandoc converts the HTML, but I'm sure anchor jumps can be used and you can write HTML tests in the browser.

      1 Reply
    • Siysuy

      I appreciate your resonse, but it is not only for complete batch export for users to turn away from siyuan, but rather for collaborative reasons and to publish notes. Users of Siyuan can collaborate and share specific notes.

      I just want a guide on how to use anchor hash. How would I proceed after I used the feature after export, so post anchor hash. I do not quite understand what to do with the new file after export.

      1 Reply
    • 88250

      Usage scenarios I think we can try to wait for the early proposers of the requirements who participated in the discussion above to respond with suggestions.

    Please input reply content ...