Commit Graph

303 Commits

Author SHA1 Message Date
Tim Abbott 712d931350 Remove banish_broken management command.
The bug it addresses is long fixed, and it's buggy.

(imported from commit b198a30f56f950168e3662dcea2bab7d908631f9)
2013-04-02 12:07:08 -04:00
Tim Abbott 2fcbb31dcf Access the UserProfile's new is_active field rather than User's.
(imported from commit ed5bdaf3e3d6d20bfb741efdac48d30482ab9ef7)
2013-04-02 12:07:08 -04:00
Tim Abbott 1443edce00 Access the UserProfile's new email field rather than using User.
This is preparatory for stopping using the User model.

(imported from commit a1b0808c8cc2ddd19a25163f91c4f18620c9ce90)
2013-04-02 12:07:08 -04:00
Tim Abbott 3a35c5b00c Add management command to logout all users.
This is intended to be used logging out users during our deployment of
the UserProfile merge, but it could be useful for other things too.

(imported from commit bfe896d854f997f7a4d06e5bc0f19ec5b1aa5e69)
2013-04-02 12:07:05 -04:00
Tim Abbott 303e77c07d Fix clearing of user sessions.
Previously, we weren't clearing the users out of memcached (we just
killed them in the database), so in fact users were not logged out
when we deactivated them for an hour (when the memcached caches would
expire).

(imported from commit 0f0a2f70e003c184106c73b22b876f57c1ef3371)
2013-04-02 11:56:31 -04:00
Tim Abbott b82edb6fd6 [manual] Add User fields to the UserProfile model.
And keep the fields updated, by copying on UserProfile creation and
updating the UserProfile object whenever we're updating the User
object, and add management commands to (1) initially ensure that they
match and (2) check that they still match (aka that the updating code
is working).

The copy_user_to_userprofile migration needs to be run after this is
deployed to prod.

(imported from commit 0a598d2e10b1a7a2f5c67dd5140ea4bb8e1ec0b8)
2013-04-01 14:34:25 -04:00
Tim Abbott 7bae0bb75a populate_db: Fix User/UserProfile confusion in personals sampling.
We were incorrectly using User objects, rather than UserProfile
objects, for fetching Recipient objects for generated messages.

(imported from commit c3dfe52f4e0a68400e22ca49293b5bf2d6986402)
2013-03-29 16:18:38 -04:00
Tim Abbott 9317035fc7 Add hashed_password argument to do_change_password.
This way we're not directly manipulating user.password() in random
management commands.

(imported from commit e6e32ae422015ab55184d5d8111148793a8aca36)
2013-03-29 16:18:38 -04:00
Tim Abbott 80747567c0 Remove test_deactivate.py management command.
It is a less-featureful duplicate of deactivate_user.py.

(imported from commit d40fe8cc30284745aa267601b4855572bbe0e26a)
2013-03-29 16:18:38 -04:00
Tim Abbott 198480ef99 Use get_user_profile_by_email more consistently.
The previous situation was bad for two reasons:

(1) It had a lot of copies of the code, some of them missing pieces:
  UserProfile.objects.get(user__email__iexact=foo)

This was in particular going to be inconvenient since we are dropping
the __user part of that.

(2) It didn't take advantage of our memcached caching.

(imported from commit 2325795f288a7cf306cdae191f5d3080aac0651a)
2013-03-29 16:18:38 -04:00
Tim Abbott 2223b60dc0 Switch functions like do_activate_user to accept UserProfiles.
Only a few of them took a User as an argument anyway.

This is preparatory work for merging the User and UserProfile models.

(imported from commit 65b2bd2453597531bcf135ccf24d2a4615cd0d2a)
2013-03-29 16:18:37 -04:00
Zev Benjamin c6df5af84d Import tornado ioloop directly instead of passing it from runtornado
(imported from commit ce901d7daca81bb1682827175bdcd83d3e5c2344)
2013-03-28 16:57:48 -04:00
Zev Benjamin e4fba59538 Make event queues persistent across Tornado restarts using cPickle
(imported from commit 1434d1e9d394d725827b1740c0c07249d5e716ed)
2013-03-28 16:57:48 -04:00
Zev Benjamin 401fa6063e [manual] Add get_events URL routing
The new nginx configuration file needs to be copied to
/etc/nginx/humbug-include and nginx needs to be restarted when this
commit is deployed.

(imported from commit 6c43f3c2c7a6acee6a852c672c96a38bda01dd0d)
2013-03-28 16:48:52 -04:00
Zev Benjamin 8f4eaa63ad Initial event system implementation
This version has several limitations that are addressed in later
commits in this series.

(imported from commit 5d452b312d4204935059c4d602af0b9a8be1a009)
2013-03-28 16:48:14 -04:00
Keegan McAllister 1ecc063d0f Split out source map processing into a library
(imported from commit 345efcc703dc1049e31fd38a6a062bf39a589eb6)
2013-03-28 14:56:52 -04:00
Tim Abbott 7685e47f07 Fix Tornado idle time logging.
When we added rabbitmq usage within Tornado, we inadvertently caused
the Tornado ioloop to be initialized in runtornado.py's imports,
before we overwrote the _poll method.  The end result was that we
weren't running the our instrumented Tornado poll function.

Fix this by moving that code to its own file which we import at the
top of runtornado.py, and adding comments documenting the situation so
we don't break this in some future import reorganization.

(imported from commit 016717476f10566fef4ed2b656f29f865d2084db)
2013-03-28 14:40:49 -04:00
Keegan McAllister f42e841ffd Add a command to annotate a JavaScript traceback with source file locations
(imported from commit 1a4d656438990b1cbefbdecc5d403573d6c5dd06)
2013-03-28 12:11:23 -04:00
Luke Faraone 0d51e59fd5 Implement URLs for API redesign.
(imported from commit 2020491a737ec4c1e99a63f84eb6cfc594a2dd56)
2013-03-28 07:57:38 -07:00
Luke Faraone b63e5e3235 Refactor runtorando AsyncDjangoHandler URLs into a list.
(imported from commit f5c65f69dd2ccb5a0ba6339b28b8c486cde2c7ff)
2013-03-28 07:53:36 -07:00
Tim Abbott a4383f8264 Remove batch_bulk_create helper function.
The behavior we require is now standard in Django 1.5.

(imported from commit 509adb0bd371081a2dad7d4a04d24fcffbda7cdc)
2013-03-28 07:36:10 -04:00
Tim Abbott e2d010ef2d Fix most unnecessary database queries in huddle creation.
This saves 2 database queries per user in the huddle when sending the
first message to a particular huddle.

(imported from commit f71aa32df846fb4b82651a93ff9608087ffcaa5a)
2013-03-28 07:36:10 -04:00
Tim Abbott 36f09729e6 Run the memcached cache filling jobs in separate processes.
This will significantly improve the speed with which the memcached
caches are filled.

(imported from commit 445fec710a00b3d19b7962beddf89d4a75322eab)
2013-03-27 12:59:23 -04:00
Tim Abbott 144987e581 logs: Provide total time passed for longpolled requests.
Also improve display of times passed -- we now use display short times
in milliseconds for easier reading.

(imported from commit 08e1e7e6acbef48453080864946f7602a3395e7c)
2013-03-27 08:19:27 -04:00
Tim Abbott f3ad2d7a6b Centralize dispatch logic for which queue processor to use.
Previous we had around 4 copies of the logic for deciding whether we
should publish data via a SimpleQueueClient queue, a
TornadoQueueClient queue, or to directly handle the operation, which
resulted in their getting out of sync and buggy (see e.g. the previous
commit).

We need to add a lock around adding things to the queue to work around
a bug with pika's BlockingConnection.

I should note that the previous logic in some places had a bunch of
tests of the form "elif settings.TEST_SUITE" for doing the work that
would have been done by the queue processor directly; these should
have just been "else" clauses -- since we generally want that code to
run on development environments whether or not the test suite is
currently running.

(imported from commit 16bdbed4fff04b1bda6fde3b16bee7359917720b)
2013-03-27 08:19:27 -04:00
Tim Abbott fde2243dbc [django 1.5] Update read flags handling to use new Django 1.5 syntax.
(imported from commit af1e2608b6d6d3677bf6266980d8dc77739aa4ba)
2013-03-27 08:19:26 -04:00
Jessica McKellar 53ec385d0b create_user: ask for realm as part of user creation.
With open realms, you can't infer the realm from the e-mail address.

(imported from commit c05afa586f121a38673f9e62281cf26c8590570b)
2013-03-23 00:19:37 -04:00
Jessica McKellar f87ffd0703 create_user: fix missing spaces in options error message.
(imported from commit 25ea284b17dfb3160aca336e31203991f70994d8)
2013-03-23 00:19:37 -04:00
Jessica McKellar e5f1dad409 create_user: remove unused imports.
(imported from commit b2dd10f73fb0285c2c92ab56b1111cf9e859cedf)
2013-03-23 00:19:37 -04:00
Jessica McKellar ea11ce4ae6 create_user: Add Terms of Service warning.
(imported from commit bdb31aebad397e98db77d3ea95ef17ac9dd4737f)
2013-03-23 00:19:37 -04:00
Leo Franchi f64bc59dfb Automatically register consumer once asynchronous connection is completed
(imported from commit 3d860a9b79a42beffbd3b73f49aa2c25986dd3c6)
2013-03-22 15:55:49 -04:00
Leo Franchi c346abe6e2 Send Tornado callback notifications via RabbitMQ
(imported from commit 83814d8c6b78fe725aedea9d98fb588ed31123e7)
2013-03-22 15:55:40 -04:00
Keegan McAllister 23b8833ec5 Handle Tornado callback notify in a more generic, less HTTP-centric way
(imported from commit 85a74d1b40461236c4c95ad688e9796ab50f0bbf)
2013-03-22 15:51:23 -04:00
Keegan McAllister 591816d80e Use TornadoQueueClient in runtornado
(imported from commit cc4d5505987dc2826d177ff93d037b17f78072b8)
2013-03-22 15:39:07 -04:00
Leo Franchi 7837e36d0c Clean up some code by consolidating
(imported from commit 3978da70286174ad3666d08bcc8db2eb28446399)
2013-03-22 09:49:54 -04:00
Jessica McKellar 4f1bf1b2f2 realm_stats: restrict some usage stats to only active users.
They are more meaningful this way -- the fact that your bots that
never log in or inactive users don't have colored streams shouldn't
impact that statistic.

(imported from commit b39debda338cbbad06957bc969b42862a888026a)
2013-03-21 17:38:20 -04:00
Jessica McKellar 85086b6662 realm_stats: Also print enter-sends stats.
(imported from commit 34f9d9916e11ac2e2748ecbb8eb4f9a34a110340)
2013-03-21 17:38:20 -04:00
Leo Franchi 6d6c3364dc Rewrite message flag handling to fork out to a subprocess for batch handling
(imported from commit 1ef846f542950cabf32f8b176f5591cf5794a0ff)
2013-03-19 09:55:12 -04:00
Jessica McKellar 1ca2f6fa7f Add a management command to reset your stream colors to the default.
(imported from commit f6891ad40088bf34686a7d8a2d910a9a0f3be7c2)
2013-03-18 15:49:39 -04:00
Tim Abbott 722918aa61 Use humbug_finish() wrapper for finish() with required arguments.
I think this is a little cleaner than the previous approach.

(imported from commit fa6233721f43c94003de643d7801a88a156affd2)
2013-03-15 12:53:07 -04:00
Tim Abbott 419e153422 runtornado: Mark block of get_response modified from the original Django.
(imported from commit 383d6960f0043175c026956e1724ec1d1157767b)
2013-03-15 12:53:06 -04:00
Tim Abbott ade7146cd9 Disable tornado request logging in favor of our Django logging.
(imported from commit 2144b2ac1cdb6062bb32fca4cc69ea3f4570cf69)
2013-03-15 12:53:06 -04:00
Tim Abbott 4df6736225 Call the Django response middleware from Tornado views.
But discard any changes the Django response middleware may have made
to anything other than the content.

This allows us to, for example, output our nice database query logging
for get_updates requests.

(imported from commit e1d2fd38ceb4d73ff50bdfaad7c72ddb24d0fe16)
2013-03-15 12:53:06 -04:00
Tim Abbott 549577a928 Pass the original request object to Tornado's finish().
This is preparatory for running the Django response middleware on
our Tornado responses.

(imported from commit 05da8ea9cb663a928b2f98a928f3992aae4f067c)
2013-03-15 12:53:06 -04:00
Zev Benjamin 22457277f4 Move send_with_safety_check functionality into AsyncDjangoHandler
This will mainly be useful in the event system branch, where we want
to actually send a response from a file other than tornadoviews.py.

(imported from commit b7ae9bb9b062215ab44eb5f0a3a72d6baeee1d07)
2013-03-14 18:48:15 -04:00
Tim Abbott 76837ebe2c fill_memcached_caches: Also fill the caches of User/UserProfile objects.
Since we flush memcached when we do a server restart, the flurry of
get_updates requests that fly in afterwards are all cache misses for
getting the User/UserProfile objects, so Tornado ends up spending
around 70ms per get_updates request rather than the usual 1-2ms.

So this should substantially improve our Tornado performance around
server restarts.

(imported from commit 07b8126bdfd4ff14e4c3362f9eda1fe5fd571c5b)
2013-03-14 15:07:42 -04:00
Tim Abbott cf58ee833a Rename lib/message_cache.py to lib/cache_helpers.py
(imported from commit 591e427aee47aea58b50d6c3fbd6a05cfe9378ea)
2013-03-14 15:07:41 -04:00
Tim Abbott 60c3cf826d restart-server: Fill memcached caches after clearing the cache.
Our previous code could in theory end up clearing the caches it had
just filled, if Tornado's cache filling work happened to be faster
than the memcached flush.

(imported from commit 48174aadad398fb7a7c917a1df765c1261b12a55)
2013-03-14 15:07:41 -04:00
Leo Franchi 0f68411859 Mark messages as read when creating them in populate_db
(imported from commit 6bf0d5bf889e94618c355e56f145b353f7574e60)
2013-03-13 14:14:46 -04:00
Leo Franchi 153983a933 Add a manage.py command to add read flags up to the current pointer
This is required because our migration is going to go in two phases.
When we do the database migration (on pushing to master), we update
all messages at that point. But prod doesn't know about the new
flags field, so any new messages sent on prod will not have the
read bit set.

When we push to prod, we want to re-run the bit of the migration script
that automatically sets read flags on messages older than the users's
pointer.

(imported from commit 961d33e972eac9ada80089bf1b1269c7fb42d56b)
2013-03-13 14:14:45 -04:00
Leo Franchi 93a3f14c43 Add backend support for handling new 'read' message flag
(imported from commit 6194e9332caa2d279cbc304f0d6a69f969aa9a72)
2013-03-13 14:14:45 -04:00
Tim Abbott a22c5e63a8 create_user: Remove unused import.
(imported from commit 7a2a7c171b87d8098c0a589c1e6228c0b131eba1)
2013-03-08 17:24:33 -05:00
Jessica McKellar 2f644f8439 Add a management command to clean up lingering tutorial streams.
We now clean up the stream subscription in more places, but some
historical tutorial streams are still around and if an error or page
reload happens during the tutorial it'll stick around.

(imported from commit 8cf0ebda26bf752c1a23296a4ba85d194bbb3004)
2013-03-08 17:01:11 -05:00
Tim Abbott acab720502 Add support for making Tornado dump its stack via SIGUSR1/2.
With this change,

pkill -SIGUSR1 -f runtornado

will dump the stack and SIGUSR2 will enable an interactive debugging
session.

This fixes #613 for Tornado which was the original motive for that
ticket; I'm not sure whether we want to do this for our Django
processes as well, but it would be easy to do so if we did.

(imported from commit a7de7c6070f4bf0404bed6f434e6a6b291d66a26)
2013-03-06 14:19:32 -05:00
Jessica McKellar ac305ffc1d Add a management command to deactivate a user.
(imported from commit 14ef58bdd2cd20c05c68cd53cf911711d3bdb5fd)
2013-03-06 11:57:04 -05:00
Jessica McKellar 30c7bdf018 Fix bug in colorize_streams that tried to include non-streams for colorizing.
(imported from commit 286a6ab38513303968b3c476ebd528a3ab2cacf1)
2013-03-04 16:30:23 -05:00
Tim Abbott 1e9a85ff05 Fix use of case-sensitive comparisons on email addresses.
(imported from commit d420169640a9f9c034b3d9ded207e583691f6652)
2013-02-28 17:49:57 -05:00
Jessica McKellar a03a639cce Add a management script to set stream colors for a domain.
(imported from commit 186e8226b57d385bbbed756615c0c63315c9d463)
2013-02-28 16:41:44 -05:00
Luke Faraone 40ad84b363 Introduce script to bulk-create activation URIs.
(imported from commit 12435d8f0d48833f52a699c0d8a0c9ec0750e6af)
2013-02-28 14:34:18 -05:00
Waseem Daher 163c9c8d75 Add a JSON call that causes the tutorial bot to send you a message.
The idea here is: part of the onboarding tutorial is going to
be you talking to the tutorial bot and it talking to you, from
our Javascript.

The reason it's driven by Javascript is that then in principle we can
do nice stuff like making popovers appear in places to point things
out to you, whereas if we were to do it strictly server-side, doing so
would be a lot harder.

The downside to doing it in Javascript is that you don't get any of
the Markdown rendering, since that happens on the server. So instead
we add this call where you give it a message, and it responds by
having the tutorial bot send you that message.

I don't think there are any security concerns here because
(1) The bot only messages you -- so you can't use it to make someone
    else think that the system is telling them to do something
(2) If there were an issue associated with having the server parse
    arbitrary Markdown, you could just trigger the issue by sending
    a message yourself.

(imported from commit b34f594dab6be6bcb81899278ae1cbe447404468)
2013-02-20 23:04:49 +00:00
Leo Franchi 49328c2551 Unify rabbitmq usage to use only one queue.
To work around the issue we're having with queue draining between
parallel blocking connections, use the same rabbitmq queue for both
activity and presence events, keyed on a 'type' flag in the message
itself.

(imported from commit 188e8fda1695734e52c5740db2195072cfc81479)
2013-02-15 11:43:17 -05:00
Leo Franchi d4a582fa1c Handle SIGINT and SIGTERM in process_user_activity by closing communication gracefully
(imported from commit dcf813fa8af1aaf5d6cc635268757e40c6c202ab)
2013-02-15 11:05:31 -05:00
Jessica McKellar 7956f859b2 management: add a remove_users_from_stream command.
(imported from commit 4aeefcc5160a382b13f8bcc2321e76eeea55f4d3)
2013-02-12 11:49:22 -05:00
Jessica McKellar 645e7d58be management: fix add_users_to_streams given do_add_subscriptions output change.
(imported from commit bc03ee0bd601122689e2062b6963780743ac7d27)
2013-02-12 11:49:22 -05:00
Leo Franchi 0a0c4bb9a0 [manual] Use rabbitmq for asynchronous presence updating
Note: When deploying, restarting the process-user-activity-commandline script is needed

(imported from commit 63ee795c9c7a7db4a40170cff5636dc1dd0b46a8)
2013-02-11 18:05:57 -05:00
Jessica McKellar 53f0b00785 Add a management command to change and log user emails.
(imported from commit 0bf25433a371f7e46ade96019f4a9ceebe395fe5)
2013-02-10 16:45:25 -05:00
Keegan McAllister d349de903b queue: Get rid of lazy initialization in SimpleQueue
Instead make it a singleton with a get_instance() class method.

(imported from commit e32cabf77b43361e74a11a23bba3a6d9fb32f82f)
2013-02-08 13:57:22 -05:00
Keegan McAllister f14db52f22 queue: Rename SimpleQueue to SimpleQueueClient
It doesn't represent a single queue; that's what the 'queue_name' arguments are
for.

(imported from commit 8382ae6793ca7c3ae4b5aacf2c128113a7a88fb6)
2013-02-08 13:57:22 -05:00
Keegan McAllister 5be28bbbb4 expunge_logs: Abort on keyboard interrupt
(imported from commit b1fe93859c15e447c0935fcfb0e83a200bc016ba)
2013-02-07 11:34:49 -05:00
Jessica McKellar 4ac4e41b30 Add a realm_stats management command.
(imported from commit e23c9c521dff5e551615977ec5cd575bb5899ec6)
2013-02-02 08:24:21 -05:00
Jessica McKellar 393fe08eb5 Add a user_stats management command.
(imported from commit 1b13fd70dc271509cf0b5524878300d17c20f50e)
2013-02-02 08:24:06 -05:00
Tim Abbott 72bc299af2 Add tool to analyze fraction of Zephyrs sent using Humbug.
(imported from commit b491961b21e845471b1c52eae2b7069cc5328103)
2013-02-01 13:37:40 -05:00
Leo Franchi 6a755c2126 Convert to []-style list comprehensions to quiet pyflakes
(imported from commit d81ec98552846d8af71394c7c530aadc53f77127)
2013-01-30 15:24:49 -05:00
Zev Benjamin 526abfccc3 [manual] Add Humbug notification bot user
The production database will need to have this user created before
this commit is pushed

(imported from commit cc8356d8afa0f0747486b7b4c82337c60499d3fd)
2013-01-15 17:02:38 -05:00
Keegan McAllister b5a0147e26 Log events to a file named after today's date
We need this so that we can safely expunge old events without interfering with
the running server.  See #414.

(imported from commit 4739e59e36ea69f877c158c13ee752bf6a2dacfe)
2013-01-15 14:37:36 -05:00
Tim Abbott e592e71515 [manual] Use rabbitmq queue to process UserActivity.
Before this is deployed, we need to install rabbitmq and pika on the
target server (see the puppet part of this commit for how).

When this is deployed, we need to start the new user activity bot:

./manage.py process_user_activity

in the screen session on the relevant server, or user_activity logs
won't be processed (which will eventually result in all users getting
notifications about how their mirrors are out of date).

(imported from commit 44d605aca0290bef2c94fb99267e15e26b21673b)
2013-01-14 13:28:23 -05:00
Tim Abbott 47b2b1cb1c tornado: Fill the message memcached in a child process.
(imported from commit 3a7b4c0f4aad37319f4cfa7892709e99883f6a87)
2013-01-14 12:28:51 -05:00
Tim Abbott 3b7d61e45f tornado: Get User and UserProfile objects from a memcached.
This commit has the effect of eliminating all of the non-UserActivity
database queries from the Tornado process -- at least in the uncached
case.

This is safe to do, if a bit fragile, since our Tornado code only
accesses these objects (as opposed to their IDs) in a few places that
are all fine with old data, and I don't expect us to add any new ones
soon:

* UserActivity logging, which I plan to move out of Tornado entirely

* Checking whether we're authenticated in our decorators (which could
  be simplified -- the actual security check is just whether the
  Django session object has a particular field)

* Checking the user realm for whether we should sync to the client
  notices about their Zephyr mirror being up to date, which is quite
  static and I think we can move out of this code path.

But implementation constraints around mapping the user_ids to
user_profile_ids mean that it makes sense to get the actual objects
for now.

This code is not what I want to do long-term.  I expect we'll be able
to clean up the dual User/UserProfile nonsense once we integrate the
upcoming Django 1.5 release, with its support for pluggable User
models, and after that I change, I expect it'll be fairly easy to make
the Tornado code only work with the user ID, not the actual objects.

(imported from commit 82e25b62fd0e3af7c86040600c63a4deec7bec06)
2013-01-11 16:11:07 -05:00
Tim Abbott 1a82741650 Move action functions from models.py to zephyr/lib/actions.py.
(imported from commit 9d577dd53ce7d4c9faf6cc8a56129d684a50811b)
2013-01-11 16:11:03 -05:00
Tim Abbott 761b7cd5fa Rename zephyr.lib.time to zephyr.lib.timestamp.
Otherwise one gets:

AttributeError: 'module' object has no attribute 'time'

when trying to use the time module from inside zephyr.lib.

(imported from commit 645368672a3eff68320278dd480edeed56721fcc)
2013-01-11 15:58:19 -05:00
Keegan McAllister 0fdc3b772e runtornado: Print busy percentage at most once every 5s
(imported from commit 017bd01f96689f06c6d166bea47b5112fad05f8c)
2013-01-10 17:21:24 -05:00
Keegan McAllister 1d9bcf8da9 runtornado: Report the percentage of time spent outside the event loop
Note that on local dev servers, this will print out every half second, as
Tornado polls for file changes for autoreloading.  In production it will only
print out when network events occur.

(imported from commit adfe88879e4e446b7dfa6ee69e0a9ad013e9c4d4)
2013-01-10 17:07:46 -05:00
Tim Abbott 3c10a2e6a0 Move bulk_create_* methods to bulk_create.py.
(imported from commit 30d5f86209f9c8807516f0440276fafeb6590c1a)
2013-01-10 16:59:36 -05:00
Keegan McAllister aa1df57163 runtornado: Drop call to tornado.autoreload
tornado.web already does this, based on the setting of the 'debug' kwarg.

Dropping this in production saves us waking up twice a second to stat()
a bunch of files.

We already explicitly restart the server on deploys.

(imported from commit 283bb0da609acb2699a04111a74c13224fe5124c)
2013-01-09 15:50:56 -05:00
Jessica McKellar 95e2948c8a management: add a script to add users in a domain to streams.
(imported from commit e234fe4d283882a5cb2e7a57bcaf8939d973b128)
2013-01-09 11:56:11 -05:00
Tim Abbott 39501e5a78 Make the Django-based test suite much faster.
So, I got annoyed that our test suite was taking forever to run:

real	2m13.443s
user	1m32.630s
sys	0m3.748s

Some quick profiling determined that the test suite is spending all of
its time loading the fixtures files (zephyr/fixtures/messages.json)
that it loads for each test case (3s to load that for each test case).
To improve this situation, I cut out from the test database used by
the test suite most of the users, subscriptions, etc. that aren't
being used directly by the test cases.  The impact is a quite
significant speedup:

real	0m15.176s
user	0m9.161s
sys	0m0.508s

We're still spending over a quarter of a second per test, which isn't
great -- but this is at least no longer unbearable.

This commit doesn't make any changes to the populate_db output if you
don't pass the new --test-suite option.

(imported from commit 2334ba5399b33edab3d29ff269fde4ea77ccd48e)
2013-01-08 17:23:55 -05:00
Tim Abbott 5601086575 populate_db: Remove obsolete mit_subs_list.
(imported from commit 601cde1ae95d3e262be8444ded48f1fdca489f7b)
2013-01-08 17:23:22 -05:00
Tim Abbott 6cd49fbb14 populate_db: Add Humbug error bot.
This is needed to avoid exceptions trying to do internal_send_message
in any test against a simple populate_db database.

(imported from commit 36927f57cbbb7e30ae249b5f1a0549fb352827f5)
2013-01-07 15:06:28 -05:00
Tim Abbott 95f4087a06 Fix dump_pointers on last time.
(imported from commit c16686492e5acd699976268cb48c0290099688cc)
2012-12-15 09:28:22 -05:00
Tim Abbott b2755472bb populate_db: Fix unnecessary import from zephyr.views.
Importing zephyr.views here has the unfortunate side effect of
creating Client ids 1 and 2 automatically (via decorators.py
instantiating the two client objects it makes), before we go ahead and
delete all objects in the database as part of the populate_db startup.

(imported from commit da03cb7606334d5926e42f422ab94d1c884937b9)
2012-12-14 17:15:32 -05:00
Tim Abbott dc2057ff73 dump_pointers and dump_passwords: Add error handling for deleted users.
(imported from commit 556c29133a4734014d4640d875c6d6fccc118668)
2012-12-14 17:15:32 -05:00
Tim Abbott ce40de14c9 Add a tool to save/restore user passwords.
(imported from commit 59bea04308bc0c42aeef840aa923dbef1919d31d)
2012-12-13 17:10:03 -05:00
Tim Abbott 434765d810 populate_db: Close database connections before shutting down.
This avoids some messages being logged by the postgres server.

(imported from commit 86aaa025a36dad6ec14d49b629cfb1ec326f929c)
2012-12-13 17:09:35 -05:00
Tim Abbott e330be7fc1 dump_pointers: Use timestamps, not IDs, for transmitting the pointer.
(imported from commit 9b51b9edc5b0bdfb2305c12a123d554811ec9e06)
2012-12-13 17:09:35 -05:00
Tim Abbott bda420a0f9 Fix email case issues when restoring user pointers.
(imported from commit 84d3288dffc1cb010d8cd2a749fe71aa2a4d0df3)
2012-12-13 17:09:34 -05:00
Tim Abbott 899cbabe23 populate_db: Make some more bits of memory possible for Python to free.
(imported from commit 2d8184d05f622475ffab1043a40251644a88c230)
2012-12-13 17:09:34 -05:00
Tim Abbott cfc3781841 populate_db: Fix StreamColor restore.
Previously, the StreamColor restore code didn't properly account for
the fact that most user subscriptions were in pending_subs and thus
not yet in the database.

(imported from commit 2e28c5a68aa045494b9336d7114c23f5c3706c28)
2012-12-13 17:09:34 -05:00
Tim Abbott 7707690231 Decrease memory usage in bulk_create for UserMessage objects.
By processing UserMessage objects in batches as we go, this avoids
consuming a large amount of memory that is linear in the size of the
messages log.

(imported from commit 0c42d97f0863da9c079836c60bebcbaeec59f849)
2012-12-13 17:09:34 -05:00
Luke Faraone 7ee945062d Add command to mark users as inactive.
This is useful when testing the sigup workflow, as this script enables you
to run through a MIT signup without manually creating a new inactive user
in the database.

(imported from commit c22649cc7c561c2fbe8682d1b17d7e5aba9ac04e)
2012-12-13 14:15:31 -05:00
Tim Abbott 1681651b7d Add dump_useractivity migration helper tool.
(imported from commit 6ceb437447c51be16b5eb0375de2f3bcf3fb14d0)
2012-12-12 15:45:55 -05:00