I exported a notebook to markdown (which is great), but I found that the files don't have links to other files like they were in Siyuan.
So, is there anyway to make markdown exports have links to other files?
Is it possible for markdown export to link to other files?
1 Reference
Related articles
-
sagar • • 2 • 1 UpVIP Warrior
@88250 I think this is an important missing feature when it comes to publishing the notes. Let me explain:
Many users, including me, like to make a lot of links between notes. We also like to publish some set of notes on our websites/blogs/digital gardens. With most other note-taking software, you can export the notes to markdown, which includes links to other markdown documents (or even specific paragraphs in the markdown documents). Then, static site generators like Hugo can ingest the markdown files and produce HTML pages for the web. These HTML pages have all the correct links to other notes' HTML pages. With SiYuan, since the exported markdown does not contain links to other notes, publishing notes with links to other notes is not possible at all. I think this is a problem for many potential users. Do you have any other suggestions for how to publish notes in SiYuan with all links-between-notes working properly?
I think this is important for another reason: With other tools like Obsidian, the notes are stored as normal markdown files. This means that the notes are not "trapped" inside the note-taking software. SiYuan does not store content as markdown files. Even if the format is still open, it still feels like the notes are "trapped" or "locked inside" SiYuan. If SiYuan has an excellent ability to export the notes to markdown (e.g. with all the links preserved) it would feel less like the notes are locked away inside SiYuan. I think this will increase trust in SiYuan, since users will not worry that if SiYuan unfortunately disappears, then their notes will disappear with it.
I understand that because SiYuan offers a lot more complex formatting options, exporting to Markdown with everything properly preserved is not possible. This comes from the limitations of Markdown. However, there should ideally be a way to publish the notes (perhaps to HTML) so that all the formatting and links are preserved. I think maybe this is currently possible for each entry in the Doc Tree, but ideally, it should be possible for an entire hierarchy in the Doc Tree i.e. a given note and all notes below it should be exported. Please let me know if this is already possible.
1 Reply1 Operatesagar updated this reply at 2023-11-29 02:41:05 -
Hello, you can try setting the reference export to footnote mode. We think this is the best ref export solution because most Markdown parsers support footnotes.
1 Reply -
sagar • • 2VIP Warrior
I think there is a fairly simple solution to this, which I will describe below. I think it can be implemented as a simple Python script that does some post-processing of the exported markdown files. I will try to create a proof-of-concept script, but it will take me a few days because I am not a programmer and will need to learn some Python programming first.
Solution description
-
In Settings > Export configure 'Ref' to 'Anchor text with block URL'. After doing this, a link to another block in a SiYuan note appears in the exported Markdown file as
[Anchor text](siyuan://blocks/block-id). -
Write a Python script that opens each exported markdown file and uses regex matching to identify all instances of
[Anchor text](siyuan://blocks/block-id). For each instance, -
Use the
/api/query/sqlAPI endpoint to get details of theblock-idby making a query like:{ "stmt": "SELECT * FROM blocks WHERE id='block-id'" } -
In the query result, you can find the
hpathkey whose value is a human readable path.- If the block-id is for a block of type document (which would be the case for a link to another document), then the Python script just needs to replace
siyuan://blocks/block-idwith hpath_value.md which would be the relative path to the exported markdown file that the link points to. - If the block-id is for a block inside a document (e.g. to some arbitrary paragraph block in the document), all you need to do is to open the exported markdown document containing that paragraph (from the value of
hpathfor that block-id), find the start of the paragraph and add an<a name="block-id"></a>anchor at the start of the paragraph. Then, thesiyuan://blocks/block-idin the exported markdown file can be replaced byhpath_value.md#block-id
- If the block-id is for a block of type document (which would be the case for a link to another document), then the Python script just needs to replace
This would basically solve the problem of having correct links in the exported markdown files and additionally the exported notes can easily be converted to HTML and published e.g. with a static site generator like Hugo.
@88250 Do you see any problems with this approach? I don't understand why SiYuan does not do this already when exporting to markdown.. but in any case, a Python script that does the above can post-process the exported markdown files from SiYuan and fix all the links. What do you think?
1 Reply1 Operatesagar updated this reply at 2023-12-01 08:25:07 -
-
-
adham • • 1Author
I was able to do it (for documents only, not blocks, as I don't know how to find out if the link is of a block or a document, and I don't want to mutate the linked documents anyway).
Here's the Python script:
import os import re import requests # Replace 'YOUR_API_ENDPOINT' with the actual Siyuan API endpoint SIYUAN_API_ENDPOINT = "http://localhost:6806" # Function to make a SQL query to Siyuan API def query_block_details(block_id): query = {"stmt": f"SELECT * FROM blocks WHERE id='{block_id}'"} response = requests.post(f"{SIYUAN_API_ENDPOINT}/api/query/sql", json=query) return response.json() # Function to process each markdown file def process_markdown_file(file_path): with open(file_path, "r", encoding="utf-8") as file: content = file.read() # Use regex to find instances of [Anchor text](siyuan://blocks/block-id) matches = re.findall(r"\[.*?\]\(siyuan://blocks/(.*?)\)", content) for block_id in matches: # Query Siyuan API to get block details result = query_block_details(block_id) # Assumes ref is of a document, not a block if result: hpath_value = result["data"][0].get("hpath") # Replace the siyuan://blocks/block-id with hpath_value.md new_link = f"({hpath_value}.md)" # Replace the old link in the content content = content.replace(f"(siyuan://blocks/{block_id})", new_link) # Write the updated content back to the file with open(file_path, "w", encoding="utf-8") as file: file.write(content) def process_markdown_directory(directory_path): for root, _, files in os.walk(directory_path): for file_name in files: if file_name.endswith(".md"): file_path = os.path.join(root, file_name) process_markdown_file(file_path) # Replace 'YOUR_MARKDOWN_DIRECTORY_PATH' with the actual path to your exported markdown files markdown_directory_path = "YOUR_MARKDOWN_DIRECTORY_PATH" # Process each markdown file in the specified path process_markdown_directory(markdown_directory_path) -
adham •Author
Here's the before and after comparison.
Before:
# Networking The most important things to know are the networking protocols. * [TCP](siyuan://blocks/20231110002225-jzlfmvr) * [UDP](siyuan://blocks/20231111114912-9gi1l0p) * [HTTP](siyuan://blocks/20231110002250-b9w413n) * [TLS](siyuan://blocks/20231111114933-oti4rz4) * [WebSockets](siyuan://blocks/20231111115839-xw23n4k) * [WebRTC](siyuan://blocks/20231111114916-nevtkga) After:
# Networking The most important things to know are the networking protocols. * [TCP](/Networking/TCP.md) * [UDP](/Networking/UDP.md) * [HTTP](/Networking/HTTP.md) * [TLS](/Networking/TLS.md) * [WebSockets](/Networking/WebSockets.md) * [WebRTC](/Networking/WebRTC.md) -
sagar •VIP Warrior
@adham Wow 😍 You are a master programmer indeed! I was still trying to figure out how to use Python to open all .md files in a directory, by copy/pasting code from Stackoverflow 😄 This solution is exactly what I meant.
I do believe it can be extended to include references to any arbitrary block, not just a document block. You did say that you don't want to mutate the linked documents, but I am just laying out the approach in case any other master programmers want to try it 😄. You wrote,
I don't know how to find out if the link is of a block or a document,
I see a few different possible checks for this
- Look at the 'type' field in the response of the SQL query. If it is a document block, the type value will be 'd'. Other type values are given in the documentation at /Please Start Here/Advanced search/Type filtering
- I believe that if the block is of type document, then the 'root_id' will be the same as 'id'. See the documentation of the SQL table fields at /Please Start Here/Advanced search/Database table
So one or both of the above checks can be used to determine if the block is of type 'document' or something else.
If the block is of type something else, then the way to link to it would be
- Open the target document that contains the block (i.e. the hpath)
- Search for the occurrence of the content in the 'markdown' field of the SQL query response. This will place the "cursor" inside the document to the start of the paragraph (or whatever other content) the block points to
- At this cursor position, insert an HTML anchor text like:
<a name="block-id"></a> - Then, update the
siyuan://blocks/block-idpart of the link tohpath_value.md#block-id
This approach enables having links in the markdown files to any other markdown files or parts inside markdown files.
@adham Thank you once again for your example code 🙏
-
adham • • 1Author
You're welcome! I actually was too lazy to write it myself so I asked ChatGPT to do it for me and changed some mistakes in its code.
And I might as well do it for non-document blocks, I will post the solution here too.
In any case, it seems like this problem can be solved easily and be integrated into SiYuan.
I am not familiar with SiYuan code or Golang in general so someone else more familiar may make a Pull Request for this feature way faster than me.
If not, I don't mind spending some time to learn enough to build this feature into SiYuan. It's the least I can do in return for getting this amazing app for free!
Thanks @88250 @sagar and long live the open source! -
Siysuy •
I noticed an issue with footnote mode posted Any export method not working (crash/freeze) app on setting -> Export -> Ref -> footnote
-
Thanks a lot for the solution. It works really. I actively write python code for my PhD. If I have time, I will for sure look into extending it with blocks.
-
@participants We are considering built-in support for anchor hash export mode Issue #10265 · siyuan-note/siyuan
1 Reply -
Siysuy •
The
anchor hashfeature currently creates a span object<span id="20231110120736-ze6m41f"></span>. Some kind of program should be able to understand and find this block inside the note as reference. Can anyone help me understand how this can be useful after export, where do I use this, which program understands this syntax?The export mode
Anchor hashfor notebook Markdown exporting has some limitations.In the case i want to publish a particular note (with links to other notes) my approach is:
- From main notebook export the notebook you want to publish to .sy.zip file
- Create new (temporary) notebook
- import .sy.zip file to new temp notebook
- Export temp notebook export it to any selectable format (only markdown is currently selectable)
Other limitation of
Anchor hasexport and the only way I got it to work (performed testing by exporting to Obsidian):- currently only works for notebook and does not allow nested notes. In the process of exporting all notes must be unnested inside notebook for .md export to function.
- Currently does not support block reference. Can only link to main note. Tested by importing to obsidian (not a siyuan issue, but a markdown syntax limitation i presume)
Depending on the responses here I might add a feature request soon that involves the concept of user selectable export syntax. In which before execution of export using
anchor hasha user can type in the conversion syntax to apply. User is presented with input fields much like thesettings > Export > Anchor text wrapping symbolfields.Conversion syntax for siyuan > obsidian would be: [[test3.md/Note 2#^d8bf62|Note 2]] and conversion input fields could somehow be [[ .. / .. #^ .. | .. ]]
1 Reply -
In theory, programs that support standard Markdown (CommonMark) should be able to support parsing and rendering inline HTML elements, that is,
<span>, but they may not necessarily support anchor hash jumps. After rendering from Markdown to HTML, the browser can support anchor hash jumping, so currently, exporting references as anchor hashs is the optimal solution.1 Reply -
Siysuy •
I think the issue remains with
anchor hashfor the reason that it is limited to notebook currently, meaning I cannot select rendering from Markdown to HTML from thenotebook_ex > export > (selectable=Markdown/siyuan.sy.zip).How would i export my notes to HTML with
anchor hash? I think I would be limited to usingpandoc input.md -o output.html, I tried it with pandoc, which succesfully creates html with link to other note, but it just opens the other note without navigation to the<span>block, so no hash jump in browser (brave/firefox) to block.1 Reply -
The reason why this function is placed at the notebook level is because some users no longer want to use SiYuan and need complete batch export, so support for this function is not considered at the document level.
I'm not sure if the span tag still exists after pandoc converts the HTML, but I'm sure anchor jumps can be used and you can write HTML tests in the browser.
1 Reply -
Siysuy •
I appreciate your resonse, but it is not only for complete batch export for users to turn away from siyuan, but rather for collaborative reasons and to publish notes. Users of Siyuan can collaborate and share specific notes.
I just want a guide on how to use
anchor hash. How would I proceed after I used the feature after export, so postanchor hash. I do not quite understand what to do with the new file after export.1 Reply
Please input reply content
...
Welcome to here!
Here we can learn from each other how to use SiYuan, give feedback and suggestions, and build SiYuan together.
Signup About