dse is a simple and crude way of not executing SQL queries in sequence, but caching values until a given max value has been met and then execute them using the executemany-method. The result can be huge speed gains.
dse was only tested on SQLite3 but intended for use in django as well.
Example usage:
import sqlite3 # for testing purposes
from dse import DelayedSqlExecutor
conn = sqlite3.connect(':memory:')
cursor = conn.cursor()
cursor.execute('create table filedata (id INTEGER PRIMARY KEY, filepath TEXT, filename TEXT, filesize INTEGER)')
d = DelayedSqlExecutor(cursor, paramtoken= '?') # using the ? paramtoken here for sqlite3. Leave it blank and it`ll use %s as support by Django etc.
d.addObject('filedata', ('id', 'filepath', 'filename', 'filesize'))
for i in range(0, 999):
# adding some dummy data. Notice the absence of the id-field. This will trigger inserts.
#Adding the id-field would trigger an update for data not yet in the db
d.addItem('filedata', {'filepath': '/tmp/', 'filename': 'test%s.txt' % i, 'filesize': i})
# No SQL has been executed yet, the default limit is 1000 items
# Adding another item will trigger the execution of SQLs and reset the d-instance
d.addItem('filedata', {'filepath': '/tmp/', 'filename': 'test%s.txt' % i, 'filesize': i})
# Adding some records to update
d.addItem('filedata', {'id': 1, 'filepath': '/tmp/', 'filename': 'testmore%s.txt' % i, 'filesize': 100})
# calling close will execute any remaining SQLs
d.close()
# you might be required to call commit on the cursor to commit the data. Depends on how you set up the cursor/connection.
What is new in this release:
- Backwards compatibility with django 1.3.x. Thanks to John Spray for this one.
What is new in version 3.2.0:
- Patch from andornaut@gmail.com to be compatible with Django 1.4.0. Patch from Herve Cauwelier to provide support for models with non-autokey primary fields.
What is new in version 3.1.0:
- Patch from rassminus; Changed sql creation to quote all references to the table name and column labels.
What is new in version 3.0.0 Beta 2:
- Fixed a few things reported by Fido Garcia.
What is new in version 3.0.0 Beta 1:
- Changes in syntax which is NOT backwards compatible, therefore a version bump. That and the nice bulk_update method.
- The add and execute methods have been removed.
- Patched models now have a property called delayed instead of dse. You can also patch specific models (new in 2.1.0).
- To insert an item call model.delayed.insert(values)
- To update an item call model.delayed.update(values)
- To delete an item call model.delayed.delete(id)
- If you have to update a huge data set where the values for the fields are limited you can use the new model.delayed.bulk_update(values), for instance metadata from photos or music files. Thanks to Cal Leeming [Simplicity Media Ltd] for inspiration on this one :-). For more info look further down for a more complete walkthrough on what happends behind the scenes.
What is new in version 2.1.0:
- Small change; dse.patch_models can now take an optional list of models to patch, like so dse.patch_models(specific_models=[User, Comment]).
What is new in version 2.0.0:
- Updated docs and examples.
What is new in version 2.0.0 RC1:
- No change in code, now released using the modified BSD license to be more compatible with Django license use.
What is new in version 1.0.2 / 2.0.0 Beta 9:
- Added FileExport-class to ease debugging what is processed during testing. Writes SQL-data to file. See source/testsuite for usage.
Requirements:
- Python
Comments not found