This should substantially decrease the amount of server load generated
by the userpresence system.
I tested that this indeed was indeed saving one query on
/json/update_active_status requests on my laptop with 2 users from the
humbughq.com realm logged in.
(imported from commit 03e9d4eb95b9f664d489862684ae162db2076e08)
It's closer to a presence query than an update, and more importantly
this moves this out of Tornado -- previously Tornado was spending at
least 3ms per recipient on messages sent to the MIT realm fetching all
this data to return back to users. This should save around 100ms per
message sent to a popular stream the MIT realm -- but more
importantly, each such event is 100ms during which Tornado is not
processing other messages.
(imported from commit 134169f0fdcd9f6640fda957edc4a28b07783d8e)
We were previously having an issue where the tutorial could
be pre-empted if you got a few messages while you were first
logging in.
I have some reservations about this being slightly fragile, and a
better approach might be to just have a bit that we use to determine
whether or not you've already seen a tutorial. (Or potentially that
checks whether or not you've ever sent a message.)
(imported from commit f8858f64a36bcd25887b76314caff283929f340c)
When narrowing, we use the selected id in the home view as the anchor,
and if the user is new and hasn't selected any messages yet (or moved
the pointer since first using humbug), their home view selected message
will be -1. Rather than failing to get_old_messages for any narrows starting
from that point, return all matching messages.
(imported from commit 72cfe392d9ac01ed41abc8eadf0f47240e374665)
I find that I never use it, and I don't totally like our
experience in the app to be different from our users'.
Admittedly, this is a small way in which that's the case :)
Finally, since we do usability studies in @humbughq.com,
the link appears there too, and I'd like it not to.
(imported from commit 1225c4ae79de52fa98b21ce00a6542df76b667ea)
Prior to this change, any stream message sent by internal_send_message
could only be in the realm of the sender.
This was a problem most notably for... the tutorial bot, with the
hilarious consequence that the tutorial worked fine in humbughq.com,
but failed to start anywhere else.
(imported from commit 33a904a28e3a57e1a2cf9172c2e2a75b50967a50)
Require POST method for /accounts/logout. This has the side effect of
automatically enabling Django's CSRF protection.
(imported from commit 44b1b6ebaadc1c03006e21ae54ac768e31234801)
But only allow them to send to tutorial-<<your username>>.
The idea being that this helps reduce potential abuse from this JSON
call. (Because otherwise, anyone could call into this endpoint and
have the tutorial bot send random messages to random peoples's
streams.)
(imported from commit 471d4348d7ad43858b5df240e4f1dceba006aab6)
After c1d98239 the function works in CasperJS as well.
Reverts some of 90f4d6ac3ddb387e74051b9af2c230698fa94479.
(imported from commit 3579df33930bb34dc081908b84900905eee6d270)
The idea here is: part of the onboarding tutorial is going to
be you talking to the tutorial bot and it talking to you, from
our Javascript.
The reason it's driven by Javascript is that then in principle we can
do nice stuff like making popovers appear in places to point things
out to you, whereas if we were to do it strictly server-side, doing so
would be a lot harder.
The downside to doing it in Javascript is that you don't get any of
the Markdown rendering, since that happens on the server. So instead
we add this call where you give it a message, and it responds by
having the tutorial bot send you that message.
I don't think there are any security concerns here because
(1) The bot only messages you -- so you can't use it to make someone
else think that the system is telling them to do something
(2) If there were an issue associated with having the server parse
arbitrary Markdown, you could just trigger the issue by sending
a message yourself.
(imported from commit b34f594dab6be6bcb81899278ae1cbe447404468)
This will discard punctuation symbols in the Postgres search and also prevent
syntax errors when users try to submit queries with symbols that to_tsquery
interprets as special syntax (such as '|' and '&').
Fixes#906
(imported from commit 3e3a0d6ae3d4a516beb8a5846f06065294ca9457)
json_to_foo will raise a ValueError if the JSON passed to it is just a
string containing a number, e.g. "1".
Traceback (most recent call last):
File "/home/tabbott/humbug/zephyr/views.py", line 711, in extract_recipients
recipients = json_to_list(raw_recipients)
File "/home/tabbott/humbug/zephyr/decorator.py", line 289, in json_to_list
return json_to_foo(json, list)
File "/home/tabbott/humbug/zephyr/decorator.py", line 282, in json_to_foo
raise ValueError("argument is not a %s" % (type().__class__.__name__))
ValueError: argument is not a list
Fixes#776.
(imported from commit 0c123a610c009eda9004cf0b0b53d60695c4e8d5)
The purpose of the validator is to ensure the user isn't active, so
let's correctly test for that here.
(imported from commit 772ddb901098f78750efab274405a10f36c49232)
Previously we checked and bailed when there was a user registered with
an email address, regardless of active status.
This meant that MIT users who had inactive accounts autocreated had
issues where they would be confusingly told they were signed up even
though they had never taken any action on our site directly.
Now we instead check whether there are any current *active* user
accounts with that email address, and proceed with generating an
activation link if the user lacks a corresponding active account.
Security implications of this commit come into play if we start
implementing removing users ability to sign in as deactivation. Since we
lack a user removal story here, this isn't terribly concerning yet and
we'll revist this code when we decide to add such functionality in the
future.
This resolves trac #581 and #631.
(imported from commit c3fb93ce065e63e19b41f63c1f27891b93b75f86)
The previous select_related didn't properly get the User object,
containing the email address, and thus would make one query per user
with presernce information.
(imported from commit 3341bc5a65387030fa8737b03ca43f79089ef56b)
On my dev machine this cuts /activity load time with lots of users by more than
2/3. I expect the gains will be even greater in production due to the greater
relative cost of database queries.
(imported from commit 0391cb29f66b618b4b99902d9fb9ab0a6cff0cb3)
Note: When deploying, restarting the process-user-activity-commandline script is needed
(imported from commit 63ee795c9c7a7db4a40170cff5636dc1dd0b46a8)
Adds a new db table for storing presences, and an API for setting
an individual user's idleness as well as fetching all idle status
for all users in a realm
(imported from commit 5aad3510d4c90c49470c130d6dfa80f0d36b0057)
Note that the tsvector cache column should be fully populated before
commit is deployed. Otherwise, full text search will be broken until
it's populated.
(imported from commit 23c36fb7d146c289148e8243c3d6a9a6494cfc62)
This allows us to remove fetch_colors() entirely, and should speed up page
load a bit.
We also JSONEncoderForHTML instead of dumps so that the result is safe
to embed.
(imported from commit 013630911960e2ac1d0bae6f5df31ad342750594)
This will give us flexibility in the future to add new properties to the
list.
In order to support that, we now do a list comprehension rather than just
returning the gather_subscriptions list in get_stream_colors.
(imported from commit a3c0f749a3320f647440f800105942434da08111)
Previously we checked if property was false after doing .strip(). Since
you can't call string methods on a NoneType, we were 500ing.
The code now does a normal dictionary get via [] and catches the
KeyError.
(imported from commit da7f28febf0865f44e92bcac1791f817c3d370f3)
Returning json_error inside an inner function call will result in the error
getting lost.
(imported from commit fd7754b15f7b62fd6e4197fd72ae03d6996a93da)
Previously we made calls to the JSON api, which means that the API key
was being ignored.
(imported from commit 46d8d0e5ac7926e824f300fd846ec42bc939e2c0)
Before, it let you subscribe only yourself or one other person to
possibly many streams.
This is used by the subscriptions page to specify the subscribers when
you create a stream.
(imported from commit c1055e98b0bb27799ac9e6dad80b9c9fd87deca2)
On my laptop, this cuts search time from several seconds down to
a few hundred milliseconds.
If we want even more speed, we could store the ts_vector as a column
on the message database. The Postgres documentation says this will
make "searches [...] faster, since it will not be necessary to redo
the to_tsvector calls to verify index matches." Going this route
requires creating a trigger to automatically insert the appropriate
column when new rows are inserted.
Note that the full text index must be fully created before this
commit is deployed. Full text search without an index is actually
significantly slower than using the LIKE operator.
(imported from commit ae74083da20d33aa2425d3e44fcdc19b160002ba)
This is for allowing us to do things more complex than returning a Q
expression, needed for doing Postgres full text search.
(imported from commit 669ec71417c04baaf8ed1774bee147079b05b03d)
Trying to add a user to an invite-only stream that already
exists will result in in error
(imported from commit 910750580a122cee92096d7e83457cb0b8cce616)
Apparently short_name is not consistently the person's username
in our system -- for many MIT people, it's their full name.
(imported from commit d975300a59c7afe5ee20b4a0db1bd51110c3df33)
As of this writing, short_name is the part before the @-sign,
and we already display the realm, so no need to display it
twice.
(imported from commit f5f9b29bff99b36b0c23176348c2869193467c38)
Eventually there should be a database table or something where we can
configure this for each realm.
(imported from commit 5f3e6823f420e113da8ff8c521cc706175299fa4)
/?lurk=foo will show all messages to the stream "foo", regardless of
whether you're subscribed.
(imported from commit 049d98b3ee8df19ef0a9dc392ae941dd463f8dd5)
Right now we mitigate abuse by sending the principal user a Humbug
informing them that they've been subscribed. Longer term, we want
some sort of stream invitation system, instead.
(imported from commit bf1f1ee0c5d3803cead1ddd9a766ff02043b629e)
The schema change made this no longer a guarantee, and creating duplicate
PUs is not harmful.
(imported from commit ca05b4dad7d3cc12a35548817f98c0a072c14c8f)
Otherwise one gets:
AttributeError: 'module' object has no attribute 'time'
when trying to use the time module from inside zephyr.lib.
(imported from commit 645368672a3eff68320278dd480edeed56721fcc)
CasperJS can't handle them; window.webkitNotifications.requestPermission()
throws a type error. We can revisit this when we want to write tests for the
notification code.
(imported from commit 90f4d6ac3ddb387e74051b9af2c230698fa94479)
This view lives at /accounts/accept_terms, and (after getting an acceptance
from the user) sends an email to all@ documenting the acceptance.
(imported from commit 8f64286ab02887fd6544fa274b2967f6499b6dbc)
Inspection of the postgres slow queries log showed that the "narrow to
personals with a particular user" database queries were taking a long
time to run (0.5s+). Further investigation determined that the OR
gate construction used here was causing the entire zephyr_message
table to be scanned; primarily I think because we were using the
implicit constraint that the logged in user had received messages.
This change makes that query explicit (improving performance), while
cleaning up the code to avoid an unnecessary query and read a little
more clearly.
After this change, the relevant database query takes 10s of milliseconds.
(imported from commit 020f5af5846c958386615e37ea9318383bf99ca0)
This essentially reverts d900957e468815bcb99de67d570dfd7ce4413220.
This code was consuming up to 50-100ms per client recipient of a
message, so for any messages that would go to 50+ browser windows /
mobile devices, it would take several seconds to run, during which
time Tornado would be completely blocked.
In the future, we can re-fix #174 using a cache of recently delivered
messages, so that this code block doesn't go to the database and thus
can run instantaneously.
(imported from commit bdfa1664210429411737f70cde54ab5a56525341)
This addresses Greg Price's feedback in #527. We now distinguish
between normal pushes, force pushes, and branch deletions.
(imported from commit 0fab6055f63ffc7e6df283b8bb8ed9971000e6d5)
feedback-bot and zephyr_mirror will need to be updated and restarted
when this is deployed to prod.
(imported from commit fe2b524424c174bcb1b717a851a5d3815fda3f69)
This makes subscribing to zephyr classes for the zephyr class
mirroring bot a lot faster, since we don't need to subscribe to the
third of our streams on which no users will actually receive messages.
(imported from commit 029b7fb260b480db5599e3c9f9effc502f6d8b59)
During long-polling mode, we were not passing the server generation
number back to the client even when they requested it. This was
causing the web client to initiate reloads every time it got a
non-immediate return from a long-polling request.
(imported from commit 66c16bf5a1c18bdd975b09d672ebdb4db8d4755d)
Previously, if the client was not interested in pointer updates, they
would still return early from get_updates() when the pointer was
updated.
(imported from commit ef9d8b5678b7e72f799840148577651ee10d47ad)
Realistically, if there is a pointer update, no other client is going
to have the same pointer. If they do, the client can ignore the
update.
(imported from commit ca2db60ed7a7c5e4b50a9fa8d350985460922a50)
We'll probably eventually get fancier with this, suggesting various
default sets depending on the company or making it easy for one person
to create default subs for several people at once.
Until we know what we want, keep it simple.
(imported from commit 14319dd50c67fe33ac6f15288dba4916ce0a89ac)
The main problem with this code is that the error message doesn't go
away until you reload. To fix this, we need to add a noblock option
on get_updates.
(imported from commit dc45af397bcf06a218bda5dd224ebd5fdf3462db)
In particular:
* Taking a list of streams as arguments.
* Using the _backend model so that we can have an API version.
* Considering "not subscribed" to be a non-fatal error.
And of course the corresponding changes to subs.js.
(imported from commit fdb300c6aa6921c2c6b09c22bd1e64405c368809)
This doesn't fully fix the problems related to not syncing
subscriptions to browser clients, but it does fix the instance that
everyone experiences.
(imported from commit be2bc31a7c4443c1678321f1a938496e2632c0d3)
This commit changes APIs and requires and update of all zephyr
mirroring bots to deploy properly.
(imported from commit 2672d2d07269379f7a865644aaeb6796d54183e1)
This should fix the problem where only one of pairs of identical
messages sent to two different zephyr classes by bots will make it
over.
(imported from commit 37005417e2e1f737501c9524b95b044eefbfe235)
These were lists of pairs because we were going to repeat keys, but that didn't
work anyway.
(imported from commit 687b3f7b8a2821d057719c725f1f39db3992ae5c)
This was causing Zephyr mirroring to break because
create_mirrored_message_users was returning False due to the
same_realm_email check failing.
(imported from commit e6a63160f34ec056461038650b5f8027718e6c63)
Personals are now just private messages between two people (which
sometimes manifests as a private message with one recipient). The
new message type on the send path is 'private'. Note that the receive
path still has 'personal' and 'huddle' message types.
(imported from commit 97a438ef5c0b3db4eb3e6db674ea38a081265dd3)
It now takes an anchor message id, a number of messages before, and a
number of messages after. The result always contains the anchor
message.
(imported from commit 84d070dc8091161c86d4bbeafbdc299493890a2a)
This was causing our tests to fail and would have also
affected API users not using our Python bindings
(imported from commit 2d81496892e9042e328279edea94be8ee4d21c1b)
return_messages_immediately's return value is not returned, so the
argument validation in @has_request_variables didn't work correctly.
@has_request_variables would return a json_error, but
send_with_safety_check expects a dict.
(imported from commit 86b6bccb7861dbf523c06b606b87374e339059a4)
This change substantially increases the number of view functions where
the API and JSON versions are actually identical code.
(imported from commit 2eee55a8943cf9a684bec2ba1f6d7afcb2b91948)
Functions with the @has_request_variables decorator can have some of
their arguments extracted from the HTTP request. For each such
argument, its default value should be an instance of the POST class.
The arguments to the POST constructor control the request variable
name that the function parameter should be populated from (it
defaults to the same as the parameter name), whether the value should
be converted before being passed, and whether a default value should
be supplied if the parameter is missing from the request.
(imported from commit ba1c25d73ba3980e44abec1458e6496807fcdaa4)
No need for an 'if' if we're just returning a boolean. And using
QuerySet.exists() should be a little more efficient.
(imported from commit 69ec3cc9f2fe904ec40ea3b8a8687a06cd03f3f3)
login_required_api_view is misleadingly named. It accepts neither a Django
login session nor login credentials (username / password). The intent here is
authentication, whether stateful (login) or stateless (API key).
(imported from commit 7e9be552168396b399116737655bd7267fd5c1a3)
We've had multiple requests from MIT zephyr users to allow
non-alphanumeric stream names, and we haven't decided what we want to
allow, so for now allow everything.
Note that the web client and mirror script limit stream names to 30
characters, which is our database limit.
(imported from commit 2acb5ee04e5ee7c40031ac831e12d09d04bbb2e6)
This is what caused our server to hang when receiving certain messages
over the last couple days. It was introduced by me making in the
assumption that doing the same thing we did after validate_notify
failed was a correct way to immediately return from
notify_new_message, which it was not. The code of validate_notify
actually finished the handler in the event that validation failed,
which isn't "correct", but did not manifest in a visible problem.
The correct way to trigger an immediate response from a tornado view
is to just return the value, not call handler.finish() and then return
None.
Similarly, the correct way to trigger longpolling from a tornado view
is to either return None (or equivalently, / drop off the end of the
function) or return a generator.
(imported from commit 5b931248b4650fc88d5d68f5936a95f19e097af9)
Tim Abbott
Leo Franchi
Zev Benjamin
Waseem Daher
Jessica McKellar
Reid Barton
Jeff Arnold
Keegan McAllister
Luke Faraone
Luke O'Malley
Tyler Hallada