Opened 15 years ago

Last modified 9 years ago

#12 assigned enhancement

Implement threaded database operations.

Reported by: Ingmar Steen Owned by: Ingmar Steen
Priority: minor Milestone: 0.9.0
Component: user interface Version: trunk
Keywords: Cc:

Description (last modified by Ingmar Steen)

Currently, the user interface blocks when an action on the database is performed (update, search). By moving those operations in a seperate thread, the user interface will stay responsive. Search results should probably be inserted into the model in batches from the main thread to reduce possible deadlocks in GTK+.

Note: filed under user interface since this is mostly toolkit dependent.

Change History (18)

comment:1 by Ingmar Steen, 15 years ago

Component: unspecifieduser interface
Description: modified (diff)

comment:2 by Ingmar Steen, 15 years ago

Priority: normalminor

Would be nice to have this for 0.9.0. Not critical though.

comment:3 by Ingmar Steen, 15 years ago

Milestone: 0.9.0

comment:4 by luke@…, 14 years ago

Hi guys,

Let me introduce myself - I'm Luke Marsden, and I'm studying Computer Science at Oxford University in England. I'm a moderately experienced Python programmer, and I'm also very impressed by Methlab - it solved all my Linux audio woes (I couldn't find a gapless media player with a usable media library *and* which could cope with multiple soundcards, so Methlab fixed this with glorious simplicity by simply allowing me to use Audacious).

I'd like to enhance it by allowing the database updates to happen in the background in a separate thread. Then I reckon it would be ready for prime-time. Ideal use case: whenever Methlab indexes your music you should still be able to search live as the data is being added to Sqlite. It should also provide a progress bar, and IMHO should index alphabetically by directory name and also show the current directory being indexed. What do you think?

Please point me in the direction of any prior work that has been done on this and with a bit of luck I'll be able to provide some diffs for your current SVN tree.

How's best to get in touch with you guys? Do you use IRC?

Cheers, Luke

comment:5 by OffHand, 14 years ago

I do not think Hyriand is actively developping MethLab at the moment. I have not seen him online for quite a while now. At some point he will probably return to this project. You can always grab the source and fork it of course....

comment:6 by OffHand, 14 years ago

Maybe you can email him at: hyriand(at)thegraveyard.org

(at) = @

comment:7 by Ingmar Steen, 14 years ago

@OffHand: Nah.. Mail to that account is moved to /dev/null temporarily because of spam issues (should be solved soonish though).

@Luke: Prior work is heavily flawed and should not be used as a reference ;-)

About contacting me: The emailaddress OffHand posted is still my MSN Messenger account, you can also reach me through google talk (iksteen <at> gmail.com) or ICQ (1533610).

I agree with you on at least the technical details (mainly because I haven't thought about the userinterface side of things yet).

The 'big' problem with this is that SQLite isn't thread-safe so my idea is having three threads: 1) user interface thread (guess what happens here...) 2) scanner thread (this actually scans the file-system) 3) database thread (a worker thread that does database stuff)

The database thread would have a job queue and each job will hold a command (or ready-made query) and a callback which is called when the job has finished.

The callback will be made responsible for thread-synchronization as to keep the database thread as simple and generic as possible.

The problem is that a lot of code will probably have to be rewritten and/or refactored to fit this design.

comment:8 by OffHand, 14 years ago

Yo Hyriand. What's up? I havn't seen you online for ages! (at least on slsk) Are you alright?

comment:9 by luke@…, 14 years ago

Hey, hyriand,

It looks like you're right and we can't assume SQLite is thread-safe on UNIX. Pity.

The three threads idea is a good one and I can see it working well. However I've got limited time available and I don't think it would make sense for me to rewrite all your UI code around the callback model. But here's an idea that could get us started...

The most important usability issue is that we want to 'background' the slow library updates, so let's:

  1. Move all the scanning code into a scanner thread. Make it toggle a variable in the main thread as to whether the database is locked or not: we lock the database before a query and unlock it afterwards. Make the scanner wait if the UI has locked the database.
  1. Write a wrapper as a drop-in replacement for the places where the UI queries the database, so rather than needing to be re-written as a series of callback functions, the wrapper would enable the UI to simply wait on the database being unlocked in the main thread before we attempt to read from it. Assuming that the time the scanner spends updating the database is negligible in comparison to the time it spends waiting on disk i/o and tag parsing, the database will be free the majority of the time. The slower queries, too, presumably, will be the more complex ones coming from the UI, but having the scanner waits on these won't affect the user's experience.

I'd appreciate your thoughts on how well this approach might work since I'm sure you know the code a lot better than I do ;)

I'll add you on MSN now...

Cheers, Luke

comment:10 by Ingmar Steen, 14 years ago

(In [113]) Initial stab at threaded database operations. For now MethLab should still work the same except that the database queries are executed synchronously in a different thread. (Refs: #12)

comment:11 by Ingmar Steen, 14 years ago

Owner: set to Ingmar Steen
Status: newassigned

Ok, that last patch should pave the way to database updates happening in the background. Basically, it wraps the DB class in a threaded variant. Currently, nothing else has changed since it basically executes queries like this:

  • push query + args on a queue and wait for an event to be set
  • DB thread executes query, pushes results and sets an event
  • main thread wakes up (because of the event that has been set)
  • main thread returens query results

Things that should happen now are making the update thread run in a separate thread and making sure the update doesn't get started while it's already running. Once that's working properly, some UI feedback mechanism should be added.

comment:12 by Ingmar Steen, 14 years ago

(In [120]) Implemented background scanning (updating without blocking the UI). (Refs: #12)

in reply to:  1997 comment:13 by Vpeaknbw, 12 years ago

comment4,

in reply to:  2005 comment:14 by Imrsavme, 12 years ago

comment4,

in reply to:  2005 comment:15 by Wdgqajzj, 12 years ago

comment6,

comment:16 by James, 10 years ago

1 در روتر 1.1.1.1 ( همچنین کل area زرد ) 4 عدد LSA Type 2 وجود دارد2 advertising ruoter ها ( با در نظر گرفتن ospf priority پیش فرض ) عبارتند از : 1.1.1.2 ، 1.1.1.3 ، 1.1.1.6 و 1.1.1.8با تشکر از مطالب آموزنده شما

comment:17 by CollireOwerry, 10 years ago

I cant get enough of this blog. Sorry i have not commented til now, but im lazy. Just wanted to eventually say thank you.

comment:18 by Rayna Silverwood, 9 years ago

I like reading through a post that can make men and women think. Also, thank you for allowing for me to comment!

Note: See TracTickets for help on using tickets.