We also now separate out the times for the socket overhead, the
request service time, and the queuing delays.
(imported from commit e1683f7f28b968b86ebb701b0ac29b00ac6d67c3)
One quirk here is that the Request object is built in the
message_sender worker, not Tornado. This means that the request time
only counts time taken for the actual sending and does not account
for socket overhead. For this reason, I've left the fake logging in
for now so we can compare the two times.
(imported from commit b0c60a3017527a328cadf11ba68166e59cf23ddf)
If a user types "/me runs to the store", we put "runs to the store"
in bold after their name.
(imported from commit fbc11e99244e1c8fa1c03e4753e706957fcd449e)
The Freshdesk API is bonkers, but we do the best we can with it to
support notifications on ticket creation and ticket updates.
(imported from commit 2023622b274ef83f4e1544d0df286fe2e68581b3)
Add a method that lets us know what percentage of a huddle is
present. (We can use this later to set the opacity of huddles
in the UI.)
(imported from commit 8a2383951807d7bfbf9d730a8980d977cf23b379)
This logging is kinda excessive since it adds like 4 log lines per
recipient, so I expect we'll end up reverting it once we've debugged
the proximal issue.
(imported from commit 5e6ab3e230f32b65ad9cf0d95f20ffbc0fe7397e)
This will hopefully help with the send dialog being stuck on
"sending" as well as allowing us to not show errors to the user on
reconnect.
(imported from commit 31ee889853f348e486863073dc130cdfb4e1338d)
Clients can only have one connection at a time, anyway, so we can
just keep track of a client id, instead. This makes reconnections
easier.
It's a little funny to use queue ids for the client id, but we know
they should exist by the time the client is connecting and they are
guaranteed to already be unique and authenticatable. We will also
eventually be integrating the event system and the socket code closer
anyway.
(imported from commit 1f60e06fb16d31d6c121deafd493fb304d19a6c2)
This reverts commit c10d9c1a0d23891acce88bf8d79866c08cb75681.
This reverts commit 9259a246946cd968a8725c38ff5ef2d4b4793717.
(imported from commit 50e9e0136c2487cc63d75ae2b78df0c70a1b0be1)
This is the amount of time between when it is sent, and when it is
rendered into the user's home view.
(imported from commit 468c28e77ba16c7256c359e90ab5aacf9d497585)
The simulated people_dict in the activity.js test was not
matching the production code, and going forward, we'll want to
share the people_dict setup for all of our tests.
(imported from commit fc21a02216b9422130b9fe9c11bcf80590612844)
Activity.js now has the capability to track huddles that
come through in loaded messages and return them in reverse
chronological order by their most recent message. Right
now this only connected to a unit test, not any production
code.
(imported from commit 59957086fa2e454e5711472df091f178217aed2b)
The main Activity page counts users as active if they have either
sent a message or updated a pointer. In the unlikely event that
somebody sent a message but never updated their pointer, we were
undercounting them, if they went through send_messages_backend.
(imported from commit 5f112be87a239980c38a18c13f9cd68e90d2e905)
This should help with determining the prevalence of slow sends as
experienced by users.
(imported from commit f00797679315c928af3c87ad8fdf0112f1dfa900)
The "desktop" counts aggregate all desktop clients, but on the
Clients tab, we are only interested in specific versions.
(imported from commit eea2d8da584a6fa32fa1f3a2bae71ef5daaba738)
We only needed a transaction here to workaround problems associated
with not having database-level autocommit.
(imported from commit 240ba05a4a4a846a7ff62e6e59e403ab0d78ab11)
I'm pretty sure this code was a hack to workaround the fact that
without database-level autocommit, .get_or_create is racy.
(imported from commit bfdd476a354aee439feeaa0030ae59cc43c165b9)
This was a precursor to UserMessage.flags.read that never got used
because we decided to use django-bitfield.
(imported from commit 868754723c07ee9b85ae951aee785e571ccfef97)
I added the @wraps decorator, and I point request._query at the
function name to override the URL, since some of the internal
URLs have realm names and domain names in them. I basically
prefer the function names in most cases, so I just made this
automatic for zulip_internal functions, rather than having to
remember to address URL-vs.-function-name for every new endpoint.
(imported from commit 5583607f395be4dfae0bac31e1cdbffdf51fb3e8)
This report will eventually replace the per-realm report that is
now accessible through /activity. In order not to disrupt Waseem,
I'm leaving the old reports around until we've polished the new
ones.
The old report does 24 different queries to get per-realm user data.
The new approach gets all the data at once, and it slices and dices
the data in Python to accomodate our slightly quirky data model.
On localhost, this is a typical query:
LOG: duration: 5.668 ms statement: SELECT "zerver_useractivity"."id", "zerver_useractivity"."user_profile_id", "zerver_useractivity"."client_id", "zerver_useractivity"."query", "zerver_useractivity"."count", "zerver_useractivity"."last_visit", "zerver_userprofile"."id", "zerver_userprofile"."email", "zerver_client"."id", "zerver_client"."name" FROM "zerver_useractivity" INNER JOIN "zerver_userprofile" ON ("zerver_useractivity"."user_profile_id" = "zerver_userprofile"."id") INNER JOIN "zerver_realm" ON ("zerver_userprofile"."realm_id" = "zerver_realm"."id") INNER JOIN "zerver_client" ON ("zerver_useractivity"."client_id" = "zerver_client"."id") WHERE "zerver_realm"."domain" = 'zulip.com' ORDER BY "zerver_userprofile"."email" ASC, "zerver_useractivity"."last_visit" DESC
(imported from commit 0c71f4e32fe5a40f4496749dc29ad3463868d55e)
This page shows aggregate activity for a user on various
clients. This allows Waseem to troubleshoot things like users
switching between website and desktop, etc.
This particular page probably won't be used too much, but some of the
logic is gonna be reused in the per-realm activity pages.
(imported from commit b8c1fad5bfa45daab40954f92319f6f89a3fa433)
If you don't have a cookie or basic auth and the request looks like
a top-level page in the browser, redirect to the login page.
(imported from commit fc1bcb1080591522bd1b694664255f7049a5d443)
The register_json_consumer() function now expects its callback
function to accept a single argument, which is the payload, as
none of the callbacks cared about channel, method, and properties.
This change breaks down as follows:
* A couple test stubs and subclasses were simplified.
* All the consume() and consume_wrapper() functions in
queue_processors.py were simplified.
* Two callbacks via runtornado.py were simplified. One
of the callbacks was socket.respond_send_message, which
had an additional caller, i.e. not register_json_consumer()
calling back to it, and the caller was simplified not
to pass None for the three removed arguments.
(imported from commit 792316e20be619458dd5036745233f37e6ffcf43)
Looking at the historical data, fewer than 50% of active users have
completed the checklist, which means that it is just persistent
clutter. We also have other better ways of encouraging people to send
traffic and get the apps now.
This commit removes both the frontend UI and backend work but leaves
the db row for now for the historical data.
(imported from commit e8f5780be37bbc75f794fb118e4dd41d8811f2bf)
We now ensure `create_realm` adds you to a default deployment and that
`create_deployment` removes the old deployment association when
performed.
(imported from commit 5b94fb07b8e11332765b057dc640a5ed873ec99e)
Subclasses of QueueProcessingWorker that don't override start() will
have their consume() functions wrapped by consume_wrapper(), which
will catch exceptions and log data from troublesome events to a log
file.
We need to do a puppet apply to create /var/log/zulip/queue_error.
(imported from commit 3bd7751da5fdef449eeec3f7dd29977df11e2b9c)
The DigestWorker isn't do any batching of events, so it can
be implemented with a simple consume() handler.
(imported from commit 0560eaf1a6e510adf97448fbe8933d9903d30016)
We were using Gravatar for user avatars, but now users can
upload their avatars directly to Zulip, and we will store
their avatar for them. This removes the old Gravatar-related
interface and polling code.
This commit does not attempt to update the avatars in
messages that have already been loaded, either for the user
making the change or other users.
(imported from commit 301dc48f96f83de0136c93de57055638c79e0961)
The "Your Account" and "Notifications" boxes on the Settings
page each had their own border and their own "Save changes"
button, but they were within the same form and sending to the
same back end point.
This commit creates a separate form and endpoint for each
of the two boxes.
(imported from commit 04d4d16938f20749a18d2c6887da3ed3cf21ef74)
This prevents the ones with external side-effects (like sending real
email) from being accidentally run in dev instances.
(imported from commit 6d9861d721abb29136bfff974de01a9264051436)
Before we were removing items individually from the queue. We now
directly use RabbitMQ's queue purging mechanism.
(imported from commit 62ab52c724c5a221b4c81a967154a4046a579f84)
This commit must be simultaneously deployed on both staging and
prod0. It also requires completely taking down the app.
To deploy these changes, do:
* check out this commit at /root/zulip on postgres0, postgres1, staging, and prod0
* stop the process_fts_updates job on postgres0 and postgres1
* stop the app on staging and prod0
* do a puppet apply on postgres0, postgres1, staging, and prod0
* move the new client certificates into place on staging and app
* move the new server certificates into place on postgres0 and postgres1
* reload the database config on postgres0 and postgres1 (this might
actually require a restart)
* run tools/migrate-db on postgres0 as root
* do a deploy through this commit on staging and prod0
* start the process_fts_updates job on postgres0 and postgres1
* do a puppet apply on nagios
(imported from commit 819bdd14326c1425e2d3041a491a8ca3b9716506)
This will allow us to redirect clients to the correct local site.
To apply this migration, just run:
python manage.py migrate zilencer 0002
(imported from commit 7bd39b5f035145b6b52e1b2cb2ad5f6720d598ce)
Here we introduce a new Django app, zilencer. The intent is to not have
this app enabled on LOCALSERVER instances, and for it to grow to include
all the functionality we want to have in our central server that isn't
relevant for local deployments.
Currently we have to modify functions in zerver/* to match; in the
future, it would be cool to have the relevant shared code broken out
into a separate library.
This commit inclues both the migration to create the models as well as a
data migration that (for non-LOCALSERVER) creates a single default
Deployment for zulip.com.
To apply this migration to your system, run:
./manage.py migrate zilencer
(imported from commit 86d5497ac120e03fa7f298a9cc08b192d5939b43)
This test has been broken for a couple months, and nobody has taken
ownership of fixing it. It's always slow, sometimes it fails
randomly, sometimes it fails for things that aren't really problems,
and it's generally been way more trouble than it's worth.
(imported from commit 8080e81b226a372e763a2558f4e5668c3a4d087c)
Use rest_dispatch for upload auth redirect so it doesn't send the
long URL to user_activity.
(imported from commit ab327bbd529412e43eee6d109f8550180544dbbb)
Trac #1734
This is implemented by bouncing uploaded file links through a view
that checks authentication and redirects to an expiring S3 URL.
This makes file uploads return a domain-relative URI. The client converts
this to an absolute URI when it's in the composebox, then back to relative
when it's submitted to the server.
We need the relative URI because the same message may be viewed across
{staging,www,zephyr}.zulip.com, which have different cookies.
(imported from commit 33acb2abaa3002325f389d5198fb20ee1b30f5fa)
As it turns out, some of these tests used message IDs 1 and 2, which
Hamlet didn't even necessarily receive as the messages to update --
which meant that they previously updated 0 messages and returned
success. So those tests started failing when I added a check for not
updating anything in the update_message_flags backend -- and this
commit fixes the tests to actually update a nonempty set of messages.
(imported from commit 9034b415d4862216a266416a8e509d987050ffd7)
This has a small bug where we don't actually filter the message out of
the home view; fixing that requires adding an index on the "flags"
field of UserMessage.
(imported from commit 492c99d0a8e87b253e577be6564bec12099bd8e9)
There seems to be some sort of bug involving PhantomJS and XHR
streaming messages. When successive pages are loaded that use XHR
streaming, PhantomJS seems to think the second one never finishes
loading and therefore hangs.
(imported from commit db93b4cab816f1fdc3f3f543c9394b1cba8abedb)
We really should be setting a variable in Javascript to indicate that
we've finished loading, but this hasn't bitten us yet.
(imported from commit ee1f7c76d9f3c482561cc5c44b81537c7e9636be)
Because our authentication system reads cookies from the initial
connection attempt, several SockJS transports can't be used.
(imported from commit 34b9571225d39072985b8223fb12c43c7235841f)
New dependency: sockjs-tornado
One known limitation is that we don't clean up sessions for
non-websockets transports. This is a bug in Tornado so I'm going to
look at upgrading us to the latest version:
https://github.com/mrjoes/sockjs-tornado/issues/47
(imported from commit 31cdb7596dd5ee094ab006c31757db17dca8899b)
The gather_subscriptions_helper() does a separate query to
get emails from user_ids, and it returns an email_dict to its
caller.
This may seem like a step backward, since gather_subscriptions()
now needs to do an additional query, but there is some benefit
in passing fewer redundant emails over the wire from the DB.
The real payoff, though, will come in subsequent commits, where
we will reduce the amount of data going over the wire to the browser,
which will benefit users with slow connections.
(imported from commit bf1cc5828a4c5f68cafd052ea29a177837970206)
I am about to change the behavior of the internal API, and it's really more
important to have test coverage on the external API anyway.
(imported from commit 8a0723cbcb4ac1819a63397584aa40e69ceb827d)
Steve Howell
Luke Faraone
Zev Benjamin
Tim Abbott
Kevin Mehall
Leo Franchi
Jessica McKellar
Jessica McKellar
Waseem Daher
Tim Abbott
Tim Abbott