Candlekeep Forum
Candlekeep Forum
Home | Profile | Register | Active Topics | Active Polls | Members | Private Messages | Search | FAQ
Username:
Password:
Save Password
Forgot your Password?

 All Forums
 Forgotten Realms Products
 Forgotten Realms Software
 New Realmslore search tool: Ask Valhaeria
 New Topic  New Poll New Poll
 Reply to Topic
 Printer Friendly
Next Page
Author Previous Topic Topic Next Topic
Page: of 2

Italian Archmage Karsus
Learned Scribe

126 Posts

Posted - 12 Nov 2022 :  00:02:53  Show Profile Send Italian Archmage Karsus a Private Message  Reply with Quote  Delete Topic
Hey, guys, I've been working on and off for a year now on a tool to find Realmslore quickly. We've been using a prototype at the wiki, it's been invaluable. If anyone wants to take a look, it's here at

https://askvalhaeria.com

Basically, you input a subject and it tries to find matches in existing sources. The database on Ask Valhaeria so far only includes web sources that were put out for free by Wizards or TSR, so it shouldn't step on any legal toes. I might add search data for other books to that database later; the database doesn't actually store the books, it only stores searches it has already made inside the books, so it shouldn't be anywhere near infringement.

Right now it's ugly as sin, I've been putting off a migration and redesign. But I could really use some feedback, if anyone has two cents to share about it. Next I gotta change the way it shows links so it's a little more user friendly, and then make all of the webpage way, way prettier.

It has other functions, too: if you click "ZOOM" on a source, it allows you to restrict your searches to that source alone. That allows us at the wiki to find where in a book a specific word or sequence shows up without having to sift through the whole book by hand. It allows for multiple queries separated by bullet points, so that we can copypaste straight off the wiki. The last function it has is a plagiarism search. It finds matches for longer sequences (minimum 5 or so words), allowing us to catch plagiarism in the wiki in a blink (as it exposes us to legal action, it's central we don't let it exist for long!).

I've been wondering what else I could add to Valhaeria to make it more useful, but in the meantime, I got told you peeps would enjoy it. So I figured I'd show ya.

Karthak
Seeker

63 Posts

Posted - 12 Nov 2022 :  12:42:52  Show Profile Send Karthak a Private Message  Reply with Quote
Seems very useful, only thing I could think of adding is an index to show users what terms are currently searchable.
Go to Top of Page

Athreeren
Learned Scribe

144 Posts

Posted - 12 Nov 2022 :  15:33:51  Show Profile Send Athreeren a Private Message  Reply with Quote
Thanks, this seems very useful! However, I have tried lots of terms that give me results on the wiki but not on this platform. How does the platform search for terms? Do you have a database that was filled manually, or did you scrape the wiki?
Go to Top of Page

Italian Archmage Karsus
Learned Scribe

126 Posts

Posted - 12 Nov 2022 :  22:26:44  Show Profile Send Italian Archmage Karsus a Private Message  Reply with Quote
Thanks for your feedback!

Karthak, there's 11 million terms in the database that it has indexes for. It can't be displayed- as it stores not just individual words, but 2-word and even 3-word sequences. The list would be endless and not easy to find; it's easiest to just ask for a term yourself and see if it's there.

Athreeren, it's not wiki scraped, it's only indexing a specific subset of the sources. Later versions may include more sources; the database scrapes sources, not wiki pages. We use Valhaeria to help us research subjects and write wiki pages, not the other way around.
Go to Top of Page

Karthak
Seeker

63 Posts

Posted - 13 Nov 2022 :  02:09:00  Show Profile Send Karthak a Private Message  Reply with Quote
Ah right, I was figuring it was a short list of maybe 10,000 terms, no point in having an index even if it was possible when most search terms will return some sources.

Is there a difference between Database A and B?
Go to Top of Page

Italian Archmage Karsus
Learned Scribe

126 Posts

Posted - 13 Nov 2022 :  12:31:47  Show Profile Send Italian Archmage Karsus a Private Message  Reply with Quote
Karthak, there is no difference between the two databases. They are a holdover from the prototype, I should later remove them.

Edited by - Italian Archmage Karsus on 13 Nov 2022 12:35:40
Go to Top of Page

sleyvas
Skilled Spell Strategist

USA
11839 Posts

Posted - 14 Nov 2022 :  19:32:34  Show Profile Send sleyvas a Private Message  Reply with Quote
searched mingari - no results
searched savyels (for the entry of a bounty hunter in powers and pantheons) - no results
searched Lauzoril - lot of results, but not to anything I would have expected (i.e. nothing to Dreams of the Red Wizards, spellbound, etc...)... but don't see how you can read the results, it just tells you where to look.

Alavairthae, may your skill prevail

Phillip aka Sleyvas
Go to Top of Page

Italian Archmage Karsus
Learned Scribe

126 Posts

Posted - 14 Nov 2022 :  22:12:39  Show Profile Send Italian Archmage Karsus a Private Message  Reply with Quote
That's right, sleyvas. It just tells you where to look, but the idea is that it never misses anything- so it helps you sift that much faster.

Valhaeria right now runs on web-only stuff- and I'm working on adding links to every web-available source. I'm working on a prototype with books indexed in it, but it's not quite ready for primetime- we just use it internally in the wiki. Even then, it still only tells us where to look, it just happens to do so in a comfortable blink.

Edited by - Italian Archmage Karsus on 14 Nov 2022 22:14:56
Go to Top of Page

BadCatMan
Senior Scribe

Australia
401 Posts

Posted - 15 Nov 2022 :  02:35:07  Show Profile Send BadCatMan a Private Message  Reply with Quote
Consider it the shareware demo version.

But I've been using the full prototype version, with sourcebooks, novels, and magazines, as well as the free articles you get in Ask Valhaeria, and it has been an absolute godsend to Realmslore research for the Forgotten Realms Wiki. Rather than spending ages searching books manually, even searching PDFs en masse, we can have a list of every source in under a second. That can be way too much for a common topic or a regular word, of course, and long phrases run the risk of false positives. But for less common stuff, it's a very quick way to zero in on the relevant sources, down to within a page or so. All we need to do is find the book and open to the page.

With it, we've been able to write articles that cover every source to mention a topic, and can include some obscure lore and forgotten sources, without having to come back later when we discover or obtain something new. We can also more easily find references for unsourced pages, so we're steadily cleaning them up on the wiki.

It's a D&D-dedicated search engine and the quickest and most comprehensive resource for lore. If you're looking for some Realmslore, do consider trying out Ask Valhaeria.

BadCatMan, B.Sc. (Hons), M.Sc.
Scientific technical editor
Head DM of the Realms of Adventure play-by-post community
Administrator of the Forgotten Realms Wiki
Go to Top of Page

sleyvas
Skilled Spell Strategist

USA
11839 Posts

Posted - 15 Nov 2022 :  15:19:28  Show Profile Send sleyvas a Private Message  Reply with Quote
I must say, I like the concept. Lord knows I'm constantly using the DOS findstr and powershell's select-string to find things for work. Never tried searching PDF's though without opening them and doing a search within the book because like many things you don't get useful outputs. This will definitely help me in my realms research, as I often end up at the wiki to look up things.

Alavairthae, may your skill prevail

Phillip aka Sleyvas
Go to Top of Page

Italian Archmage Karsus
Learned Scribe

126 Posts

Posted - 25 Nov 2022 :  22:37:53  Show Profile Send Italian Archmage Karsus a Private Message  Reply with Quote
Thanks for the vote of confidence, BCM!
I've been banging my head against the code for a bit, looking to upgrade the database. In my next delivery I'll see about throwing in archive links for every web resource, the novels with Google Books, Google Reader, Internet Archive and Amazon links (all of those let you have a taste for free, I figure they'd be useful for a thrifty lore scholar), and look at getting a presentable version of the magazines somehow.
Go to Top of Page

martynq
Seeker

United Kingdom
90 Posts

Posted - 28 Nov 2022 :  16:18:57  Show Profile  Visit martynq's Homepage Send martynq a Private Message  Reply with Quote
This looks like a very worthwhile endeavour ... and far more sensible than the vast text file that I once embarked upon with every reference that I could find in every FR book. (My project was long since abandoned since it was clearly not feasible to complete and it wasn't using technology in a sensible way -- though I did start it back in the late 1990s when such technology was harder to come by!!)
Go to Top of Page

Italian Archmage Karsus
Learned Scribe

126 Posts

Posted - 30 Nov 2022 :  23:22:22  Show Profile Send Italian Archmage Karsus a Private Message  Reply with Quote
Thanks, martynq.

I've managed to finagle the SQL into adding links and topics; the next version of Valhaeria ought have links to every column it has cribbed in Internet Archive, with print friendly and one-file download options wherever available. Crawling through the IA's glacial interface is something I wouldn't wish my worst enemy... I've considered local downloads in case the IA goes down, but that'll hafta wait for next version: I don't want anyone to take my site's word for anything as it is intended mostly as a wiki writing accellerator.

I also think I've refined the crawler module to get all of the IA's pages as much as I'll be able to refine those. The rest, I figure I'm doing by hand the hard way.
Go to Top of Page

Italian Archmage Karsus
Learned Scribe

126 Posts

Posted - 06 Dec 2022 :  00:09:48  Show Profile Send Italian Archmage Karsus a Private Message  Reply with Quote
So, uh, I improved on Valhaeria some, at last. Now we have links, on almost every web source, and a few novels. I'll make it a point of adding more sources, and providing links to every one of them whenever available. This should improve its useability, even if many of the links are unfortunately broken.

I've lost some references while rebuilding- I'll get them back, I promise.

Edited by - Italian Archmage Karsus on 06 Dec 2022 00:58:47
Go to Top of Page

Italian Archmage Karsus
Learned Scribe

126 Posts

Posted - 18 Jan 2023 :  22:22:55  Show Profile Send Italian Archmage Karsus a Private Message  Reply with Quote
Hey! I fixed up the database. For whatever reason it has rejected a handful of sources, which I'll get to in due time.

Try it on now for size. It's got a single database, no more database swapping button, and it's got links to its every source. Most of them work, too.
Go to Top of Page

Italian Archmage Karsus
Learned Scribe

126 Posts

Posted - 08 Feb 2023 :  22:31:20  Show Profile Send Italian Archmage Karsus a Private Message  Reply with Quote
Hey! Twitter's about to sunset its API access, so this might be the last time I get to grab tweets en masse. Did I mention Valhaeria also links to tweets? I did my best to skip the tweets that related to his personal life or to other projects, so I might have caught some Realmslore in that blacklist... but it's more comfortable this way.

Also, recently added a new capability! By prefacing your query with ">>", you get a snippet from every page showing you exactly what text it thinks you are matching. This system uses some maths to guess what the text must've been, as Valhaeria most emphatically does not, can not, and will not ever store the full text, but it might be good for quickly ruling out turns of phrase when you're looking for something in particular.

Be warned, the system is heavily taxed by this function, I'd prefer it if you didn't use it for queries that have too many results. I'm not running this one on a super robust VPS.



Val's almost feature complete now. Just needs one more feature, and I'll call it a day. It's going to be mostly for us wiki editors, tho, so I guess as it is, Valhaeria's feature complete for general use. The next feature is going to be pointless for anyone who isn't a wiki editor: I am going to create a checker that looks for matches *outside* a given book, so that when we search for, say, "Valraxaxath" in his own article, we only get hits for Valraxaxath OUTSIDE that article. A thingy for knowing if something exists OUTSIDE a particular book, that is. :)
Go to Top of Page

Italian Archmage Karsus
Learned Scribe

126 Posts

Posted - 16 Feb 2023 :  00:41:19  Show Profile Send Italian Archmage Karsus a Private Message  Reply with Quote
Hey! Added the final feature. It's mostly for wiki editing, but by adding space-bullet point-space between search queries, you get results separated by query, instead of by year. This is useful for telling whether any of N elements after indexing requires us looking other sources before starting a fresh article.

From here on, I just gotta fix some stuff with the text digester, and add a bunch if other things from the archived versions of TSRINC.com and TSR.com. Meager, but best have them all, I figure! That's how I learned that the Realms and Greyhawk are on the same planet, for instance. We have to put this in the wiki! Everyone's gonna hate it!

https://web.archive.org/web/19970424044506/http://tsrinc.com/info/tabloid.html
Go to Top of Page

Athreeren
Learned Scribe

144 Posts

Posted - 16 Feb 2023 :  08:08:18  Show Profile Send Athreeren a Private Message  Reply with Quote
quote:
Originally posted by Italian Archmage Karsus

Hey! Added the final feature. It's mostly for wiki editing, but by adding space-bullet point-space between search queries, you get results separated by query, instead of by year. This is useful for telling whether any of N elements after indexing requires us looking other sources before starting a fresh article.

From here on, I just gotta fix some stuff with the text digester, and add a bunch if other things from the archived versions of TSRINC.com and TSR.com. Meager, but best have them all, I figure! That's how I learned that the Realms and Greyhawk are on the same planet, for instance. We have to put this in the wiki! Everyone's gonna hate it!

https://web.archive.org/web/19970424044506/http://tsrinc.com/info/tabloid.html



Any idea what this is about? "The above story is a work of fiction. Any resemblance to any being living, dead, or employee of TSR, is purely concidental." makes me thinks it's more satire than April fool, but in 1997 or earlier, I'm not sure what this is in reference to. Have you been able to find more about the context of that joke? (although I'm definitely keeping that bit about Vecna being a bartender in Waterdeep)

Edited by - Athreeren on 16 Feb 2023 08:09:30
Go to Top of Page

Karthak
Seeker

63 Posts

Posted - 16 Feb 2023 :  09:44:01  Show Profile Send Karthak a Private Message  Reply with Quote
The context is in the name, it's a parody of a trashy tabloid article.
Go to Top of Page

Italian Archmage Karsus
Learned Scribe

126 Posts

Posted - 05 Mar 2023 :  17:45:23  Show Profile Send Italian Archmage Karsus a Private Message  Reply with Quote
I'm about to take another swing at making the database a bit more complete.

For my next act, I'm going to cache the page preview (still working with free mats, shouldn't be a problem) in every page. I've rebuilt the bulk DL'er to cast a bit wider net. I'm also going to include more DOC files, and the piece de resistance, I'm going to grind up and snort the old TSR and RPGA pages. This may add some stuff that is a wee sketchy and of dubious finality, but better sketchy than gone for good.
I also gotta rework the tweet-sucking module a li'l dri'l bit. The current version's OK-ish, but sometimes it fails to hoover up the context where the search keywords are supposed to be. That's not great. Finally, there's PDF versions of game manuals that shouldn't be a challenge to hoover up.

Was also considering tossin' up a PayPal link. I'm not doing this for a living, but hey, why not?
Go to Top of Page

Italian Archmage Karsus
Learned Scribe

126 Posts

Posted - 15 Mar 2023 :  14:32:16  Show Profile Send Italian Archmage Karsus a Private Message  Reply with Quote
It took me way longer than I thought, but the new site is up!

Forgot a few sources... Will throw them in later today, my bad.
Go to Top of Page

Ayrik
Great Reader

Canada
7989 Posts

Posted - 16 Mar 2023 :  00:10:53  Show Profile Send Ayrik a Private Message  Reply with Quote
I tried using the site.

But I couldn't follow any of the links. Because the font colours are too painful. The only way I could really read them is with highlighting.

Sorry.

[/Ayrik]
Go to Top of Page

Italian Archmage Karsus
Learned Scribe

126 Posts

Posted - 17 Mar 2023 :  02:04:55  Show Profile Send Italian Archmage Karsus a Private Message  Reply with Quote
I'm open to feedback, of course. I can change the colors, shouldn't take more than a moment. ;) If you don't have any suggestions I'll just check out style guides online, how hard could it be?
Go to Top of Page

Karthak
Seeker

63 Posts

Posted - 17 Mar 2023 :  02:17:49  Show Profile Send Karthak a Private Message  Reply with Quote
The purple used for showing clicked links looks fine, for some reason the blue looks blurry except if zoomed in, maybe changing it to a bright orange or red will make it stand out more.
Go to Top of Page

Ayrik
Great Reader

Canada
7989 Posts

Posted - 17 Mar 2023 :  08:08:25  Show Profile Send Ayrik a Private Message  Reply with Quote
I'm using dark mode, basically a black or dark grey background.

That particular blue on black is somehow just bright enough to be obnoxious and just dark enough to be illegible. On my small laptop screen and on my large desktop monitor. I want to get closer to make out what it says but I can't look at it long because of the awful brightness.

Not trying to be negative. But my eyes aren't as young as they used to be, lol.

[/Ayrik]
Go to Top of Page

Italian Archmage Karsus
Learned Scribe

126 Posts

Posted - 17 Mar 2023 :  22:31:04  Show Profile Send Italian Archmage Karsus a Private Message  Reply with Quote
Alright! Lightened up lonk colors. I hope this will be more forgiving on your eyes, going forward.

Thanks for the feedback! Please let me know how it is right now!
Go to Top of Page

Ayrik
Great Reader

Canada
7989 Posts

Posted - 17 Mar 2023 :  23:08:00  Show Profile Send Ayrik a Private Message  Reply with Quote
That is much better, I can read it in the dark, I can read it in sunlight. Thank you.

[/Ayrik]
Go to Top of Page

Italian Archmage Karsus
Learned Scribe

126 Posts

Posted - 19 Mar 2023 :  02:54:40  Show Profile Send Italian Archmage Karsus a Private Message  Reply with Quote
Alright. Added a thousand and change sources to the database. It's getting pretty complete... If anything it is somewhat overfilled, some of them turned out to have failed to display links correctly and some others just don't belong.

It will take some time to go the rest of the way. There are about 70 columns missing about 310 articles and i didn't want tbem in the DB yet. I will try to stick those in next update. See you 'till then!
Go to Top of Page

Italian Archmage Karsus
Learned Scribe

126 Posts

Posted - 25 Mar 2023 :  23:04:01  Show Profile Send Italian Archmage Karsus a Private Message  Reply with Quote
Stuck in another database, hopefully with less Star Wars and Neopets and Harry Potter and other bycatch. Just enough to let us check for crossovers and inspirations, I hope!

Also added a LIBRARY view. It will let you know exactly what we've got in the base. Please don't abuse it! It's still a lot of results involved in it, and may be a wee slow for your connection.

Edited by - Italian Archmage Karsus on 25 Mar 2023 23:04:56
Go to Top of Page

Athreeren
Learned Scribe

144 Posts

Posted - 12 Apr 2023 :  12:14:50  Show Profile Send Athreeren a Private Message  Reply with Quote
The Mages and Sages podcast can dump lore at high speed, and there is of course no way Youtube's automated subtitles can properly spell the names of the uncommon locations mentioned by the sages. Ask Valhaeria has been very useful for this, and I'm wondering how I can improve my search process: can I include joker characters, or characters that could be one of a few possibilities?
Go to Top of Page

sleyvas
Skilled Spell Strategist

USA
11839 Posts

Posted - 13 Apr 2023 :  14:15:08  Show Profile Send sleyvas a Private Message  Reply with Quote
just wondering... has anyone asked something like ChatGPT yet to "write a report about Khelben Arunsun" or "make a list of known harpers from the forgotten realms". I've seen the news going on and on about this AI, but I've yet to setup an account (there is a portion of me that loathes the idea of making an account with them).

Alavairthae, may your skill prevail

Phillip aka Sleyvas
Go to Top of Page
Page: of 2 Previous Topic Topic Next Topic  
Next Page
 New Topic  New Poll New Poll
 Reply to Topic
 Printer Friendly
Jump To:
Candlekeep Forum © 1999-2024 Candlekeep.com Go To Top Of Page
Snitz Forums 2000