-
Notifications
You must be signed in to change notification settings - Fork 747
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BDTD file importer #2852
base: master
Are you sure you want to change the base?
BDTD file importer #2852
Conversation
Import search results from Biblioteca digital brasileira de teses e dissertações (BDTD; Brazilian digital library of theses and dissertations; https://bdtd.ibict.br)
Thanks, a couple things:
|
OK, looks like the huge file size is coming from the second test. A few issues contributing:
|
Hi! Actually, the last commit was a mistake. I didn't finish the web translator, when i did the pull request i had the file importer working and a not so big test file. I will try to finish the web part soon. |
This reverts commit 29cc7ce.
Ok, i reverted the last commit, because was a mistake push that to master, so now the head has just a small test case and a file importer that works. About the web importer, i said before that i would finish that soon, but i didn't knew how web importers workers, and now i see that this json importer that i did is not useful to import just a page. This JSON format is one of 2 formats options that the user can download the full search result (the other is CSV, both looks non standard), by clicking in an "export" button. It's a download option for the user, not a JSON used in the page itself. To import just a page would be better parse the HTML DOM instead. But it seems me a different use case. The user can download and import a full search result in order to make a systematic literature review. As I said, BDTD is maintained by the government and i guess it has some relevance for Brazilian researchers, because all master and doctoral thesis from a lot of brazilian universities are there. |
This should be a web translator and should support both search results and individual item pages. A site's size isn't an argument for making its JSON schema into an import translator - if that schema isn't used by more than just one site or as an interchange format, there's no point. I'd imagine there's a way to export a single item as JSON, even if it's not exposed in the frontend - is there not? |
Hi! I've found a way to that JSON only of the page visible items, and i'll try finish the web translator, in the right way, later, but i have a problem. Zotero already has some web translator that partially works with BDTD. It's importing only title, authors and year, but there is a lot of data missing (including abstract and tags). I checked there is no reference to BDTD URLs in the repository at all, but i guess that something works because BDTD site is a instance of VuFind software that is used a lot in this kind of site. The problem is I don't know how i check which translator is doing that and if can do a more specific one. * Actually, that JSON link is an API from vufind (check https://vufind.org/wiki/development:apis:search#search_api). I don't know if it can be reused with others VuFind based sites. That JSON is too weird to be a standard, it should be based in some local setting. |
Why? It seems like a standard feature according to that page... What isn't working well with the current VuFind translator? Can we just add some fixes there instead of writing a new one with a different API? |
Import search results from Biblioteca digital brasileira de teses e dissertações (BDTD; Brazilian digital library of theses and dissertations; https://bdtd.ibict.br), exported as JSON.