On my laptop, this cuts search time from several seconds down to
a few hundred milliseconds.
If we want even more speed, we could store the ts_vector as a column
on the message database. The Postgres documentation says this will
make "searches [...] faster, since it will not be necessary to redo
the to_tsvector calls to verify index matches." Going this route
requires creating a trigger to automatically insert the appropriate
column when new rows are inserted.
Note that the full text index must be fully created before this
commit is deployed. Full text search without an index is actually
significantly slower than using the LIKE operator.
(imported from commit ae74083da20d33aa2425d3e44fcdc19b160002ba)
This schema migration is only for use in automated migrations. To
deploy on the production database (the migration only needs to be
done once for both of staging and prod because they share a
database), you should instead execute the following SQL manually:
$ ssh postgres.humbughq.com
$ psql
humbug=> CREATE INDEX CONCURRENTLY zephyr_message_full_text_idx ON zephyr_message USING gin(to_tsvector('english', subject || ' ' || content));
Note the addition of the "CONCURRENTLY" keyword. The problem is that
creating the index takes non-trivial time and requires a write lock
on the table while the index is being created. This would mean that
users would be unable to send messages while we were generating the
index, which isn't acceptable. We can't create the index
concurrently in the South migration because concurrent index
creations can't happen inside of a transaction and South forces a
transaction on migration functions.
Also note that this index must be created before Postgres full text
search is deployed to the app because full text search without an index
is actually much slower than plain search using the LIKE operator.
(imported from commit 8b9445c27d0e427278de997b22342bffe6d855b7)
This is for allowing us to do things more complex than returning a Q
expression, needed for doing Postgres full text search.
(imported from commit 669ec71417c04baaf8ed1774bee147079b05b03d)
We will minify our code, rather than trying to restrict who can see the
un-minified code. Removing access control first simplifies things.
Manual deployment steps:
scp servers/puppet/files/nginx/humbug-include/app root@staging.humbughq.com:/etc/nginx/humbug-include/
ssh root@staging.humbughq.com service nginx reload
and then the same for app.humbughq.com once deployed to prod.
(imported from commit 63788aa3fa7ba5fd97fcf85b05760abb5e7cae4b)
1) Make the search hotkey (/) no longer clear the search box
2) Vertically center the "Search..." placeholder
(imported from commit 02dee6b72c4457e160f57d8604164e15e62f5f28)
This fixes Trac #723 - Message view scrolls to top after reloading on another tab (e.g. settings)
(imported from commit d9134cec6879625d577c43a08d258af3f6dacc5f)
Supporting ``` as a code fence marker complicates the auto-fence
closing, and as per a discussion with Keegan on code-review@, it
is not worth the extra complexity.
(imported from commit 405afb95c4295a02f4677181456caf9d49913ac4)
We don't use the latter anywhere in our API, and this typo caused user
API keys to be emailed / humbugged places.
(imported from commit d0402e8e9fd587f6a9018c962d222fb5f9ceca48)
Under some unknown circumstances (it happens on Keegan's laptop) the
loading indicator text width calculation returns a result that's a
few pixels too small. We hack around this by setting 'white-space'
property on the container div to 'nowrap'. The container's div will
be slightly too small (and thus the text will stick out the right
side by a few pixels), but that's probably OK.
This fixes#698
(imported from commit 2e5b28fb3a1fca78c700af74a452bfafd09a2da9)
This is to make the loading_indicator_text height closer to
loading_indicidator_spinner's height.
(imported from commit 062973e9c09b005601c5c943ce65dc97eeee179f)
This should fix the weird positioning spinners had when they were
created while hidden.
(imported from commit 777d5c376a3f93b0b7b3b749877070b438b7c102)
This changes the sorting for autocomplete to:
* Properly prioritize case sensitive and insensitive prefix
* For recipients, prefix-search on email, then sort by most recently sent pm
(imported from commit 893c7a07d678644a418a69656180fadf0d6f374d)
The fact that the user sees a change (the button changes or a row
appears) makes it obvious that the operation was a success. The
success messages were only serving to make the page scroll
confusingly when you couldn't see the top.
(imported from commit 471b9304f71bb8533c98d208b855d4d75c04a886)
The query fails anyway, so this is not a safety check. This commit
makes it so that MIT users actually see their subscriptions instead
of an error.
(imported from commit ee635943728d7d9823e118d9fa51c402b1cd9bf2)
We now keep around the subscription information for streams that the
user unsubscribed from in stream_info and have a field that
indicates whether the user is currently subscribed.
(imported from commit 973e2f4bd4139157b03d7c1a372db93a1a5130f7)
Trying to add a user to an invite-only stream that already
exists will result in in error
(imported from commit 910750580a122cee92096d7e83457cb0b8cce616)
Previously it was just testing that sending a GET request to our
POST-only URLs returned a 400 error.
(imported from commit a510734271385046bbf29166b38c0ecbd104d2da)
If get_updates requests a message ID that is too old, Tornado will
return a 400, and currently the only way to start caring about newer
message IDs is to reload.
(imported from commit 1638d71868475ffd793162afc7a2731cab14bd75)
Longer-term this should be done in a more sustainable manner, i.e.
including Glyphicons like we currently include Halflings.
(imported from commit 3f405c969352481626614a0d61874bd77f0388b7)
This commit both causes the settings button not to be drawn as
pressed, but also fixes the issue we were experiencing where,
e.g. changing from "Home" to "Settings" and back to "Home" would cause
you to lose your place.
(imported from commit 5084b280a202f6bf8f811834bf9d2734a034c8c1)
This is really the first step of implementing the "Oppa Gmail Style!"
redesign, and is largely an HTML/CSS-based change, with some
slight JS tweaks to deal with things being renamed or being no
longer necessary.
(imported from commit e05adc283ea066f0f90009cf712c4f3657c2485a)
Prior to this commit, at <979px, the .container in a .navbar has
`width: auto`, but a normal .container has width 724px, which causes
the two to drift out of sync.
This fixes that.
(Arguably, it's weird for us to waste ~200px scrunching
this down to 724px at this ratio, but we can solve that
as a separate issue later.)
(imported from commit 1f431ca1e2168db75821ea0be43941d29fd3e6b8)
We always want the navbar to stick at the top, no matter what
the screen size, and we want it to consistently look the same
height, etc. regardless of our page width.
This is possibly also accomplished via position: absolute !important
and other overrides in our own CSS, but this actually seems
slightly cleaner in a way.
(imported from commit 340fafb49bcbc1088a816897d320e252c4615d19)
Some time between the 2.0.4 and the 2.1.0 upgrade, Bootstrap broke in
a way such that clicking on a dropdown did not cause it to close.
Here's the bug thread about it:
https://github.com/twitter/bootstrap/issues/4497
I've implemented this workaround discussed there, though the bug is
fixed in 2.1.1, so when we upgrade this will go away (which is why I
only reluctantly tag it 'third', since the diff will not need to be
carried forward.)
(imported from commit f8d9cf65b33306a426d864c9b503bb3446614111)
Apparently short_name is not consistently the person's username
in our system -- for many MIT people, it's their full name.
(imported from commit d975300a59c7afe5ee20b4a0db1bd51110c3df33)
As of this writing, short_name is the part before the @-sign,
and we already display the realm, so no need to display it
twice.
(imported from commit f5f9b29bff99b36b0c23176348c2869193467c38)
This should fix the issues we saw where users who hadn't received a
message in the cache would not be able to receive new messages.
(imported from commit 3834dd5bd5a10c7a603194b540d78afd8a966017)
When not in lurk mode, lurk_stream is undefined which caused
lurk_stream.toLowerCase() to fail.
(imported from commit 63ce79083b55a37cb0455871237a76d724fbbbea)
Eventually there should be a database table or something where we can
configure this for each realm.
(imported from commit 5f3e6823f420e113da8ff8c521cc706175299fa4)
This is needed for the next commit so that the loading indicator is
created while its associated div is visible.
(imported from commit 72d6ccc14158b49e0ea640ab818114869aa548bf)
This is a hack to work around the fact that the query here takes far
longer than it should.
(imported from commit e6ee00c0636fe85cd946e4e3f2bbb2bda2da1f26)
The stream message cache query was grabbing an overly broad set of
columns and related models (e.g. the sender's user_profile), so that
Tornado would spend too long doing the query.
(imported from commit 6a692bc0e31957d4b9d9528d348c4b07a09eb15c)
As a side-effect of customizing the e-mail, this also makes the host
on which the error happened a part of the subject line.
(imported from commit 7d5e9ad108b48fd34528512c5955567119935d4e)
Prior to this commit, they weren't able to subscribe or unsubscribe
from streams or change stream colors.
(imported from commit 7f690c724bec3e7e6ba3b45ac7b41d1f7296b6f1)
If you create a spinner in a hidden element and then show the
element, the spinner is placed differently than if you had created
the spinner while the element was visible. This commit makes it so
that we never create spinners while their parent is hidden.
(imported from commit a21e68976d70fcceece30ee35f5e7cf6f9490497)
This allows us to use a uniform style across all our spinners. It
also cuts down on boilerplate HTML.
(imported from commit 9879f38e0f1ca8edd40a937753811e329447262d)
/?lurk=foo will show all messages to the stream "foo", regardless of
whether you're subscribed.
(imported from commit 049d98b3ee8df19ef0a9dc392ae941dd463f8dd5)
A public stream is one for which any user can view all messages sent
to the stream, regardless of whether the user was subscribed when
those messages were sent.
For now, to avoid a database schema change and to facilitate testing,
public streams are all streams on the customer29.invalid or
humbughq.com realms.
(imported from commit 7a71fd788d585a6f5b3e494e771ec85b632bb36e)
Before, a whitespace-only search would "highlight" the pointer and all kinds of
other interstitial space, which totally broke the page layout.
Fixes#408.
(imported from commit e7e0e251551a9da5a2ea53e36b9cce16e3e30634)
* Ignore beginning and ending whitespace when submitting the form
* Ignore beginning and ending whitespace when doing autocomplete
* Don't autocomplete on just whitespace
(imported from commit b3231e08f6797a38bafbcef2e694f4bae059c20f)
Right now we mitigate abuse by sending the principal user a Humbug
informing them that they've been subscribed. Longer term, we want
some sort of stream invitation system, instead.
(imported from commit bf1f1ee0c5d3803cead1ddd9a766ff02043b629e)
The production database will need to have this user created before
this commit is pushed
(imported from commit cc8356d8afa0f0747486b7b4c82337c60499d3fd)
We need this so that we can safely expunge old events without interfering with
the running server. See #414.
(imported from commit 4739e59e36ea69f877c158c13ee752bf6a2dacfe)
So that it is, in fact, navigable via keyboard shortcut. It's not the
most beautiful way to do this; in the ideal world the layout would be
more properly responsive and could fill up the screen for you when you
have the room for it.
The issue is that the version of Bootstrap we're running doesn't
easily allow for variable-sized modals, and the hassle of showing
and hiding a div manually didn't seem worth it.
The pixel sizes are specified manually as well, in a way that
isn't particularly responsive, because of a few competing
desires:
1) For them to appear side-by-side when wide enough
2) For them to stack when not wide enough
3) The fact that they're awkwardly between a span2 and a span3,
so we can't just use the Bootstrap responsive layout stuff
Finally, the diff here is best viewed with -w. (I also swapped the
order of Narrowing and Navigation, which unfortunately clutters the
diff.)
(imported from commit 62cfa7959d48ce515562713eb48f9d9b33c8cc16)
Before this is deployed, we need to install rabbitmq and pika on the
target server (see the puppet part of this commit for how).
When this is deployed, we need to start the new user activity bot:
./manage.py process_user_activity
in the screen session on the relevant server, or user_activity logs
won't be processed (which will eventually result in all users getting
notifications about how their mirrors are out of date).
(imported from commit 44d605aca0290bef2c94fb99267e15e26b21673b)
I'd like to phase this out in favor of something you actually
see only when you don't use Tab-Enter, but that's more challenging
than I'm prepared to tackle right this second.
(imported from commit eeda53b0ed69d0e528b00ea9e7c7a20edb35df34)
If you're currently composing a message and you click one of those, it
now no longer clears out your old message contents, making it more
analogous to the job the old selector used to play.
(imported from commit b935a3bf307bdbd82f1ee7db31d3a3c89c623195)
Accomplished by:
- Hiding the stream/PM selector
- Eliminating the "tab to send" reminder
- Moving send and formatting links to the right
- Reducing the size of the 'subject' box
- Generally tightening up whitespace
To be fixed later in this series:
- A Tab-Enter reminder
- Completely eliminating the stream/PM selector
(imported from commit 7efe04adcbe373f99a36d3ba23b32944c17aa099)
internal_send_message now has the ability to send personals as well as
stream messages.
This change was accidentally lost during a rebase.
(imported from commit 153a3929c5c64be82288293c1f0cc02fcc03c08d)
Later, we can fork off a separate process to do that task, but for
now, we can just rely on the fact that unless the server was just
rebooted, the relevant messages will already be in that cache, and
even if it was, getting the needed messages once each is not a
particularly expensive query.
(imported from commit 6d08eba1b41237fd4e1204e181ce8f227573930d)
It can easily cost an entire line, and the information is available by
clicking on the sender's name. Plus on a phone, you can't hover
anyways.
Annoyingly, this tends to put the popover partly off the window, but
there must be a way to fix that separately.
(imported from commit 19334cb067981b323e300245654c83c8e545fb2e)
It's possible for selected_message_id to not exist in zhome. For
instance, when you open the page to a narrowed URL, there is a race
between loading the narrowed messages and loading all your
messages. If the narrowed message request completes first, it will
update selected_message_id to your initial_pointer if the latter
happens to be among the narrowed messages. Even if it is not, you can
select a message by clicking on it before the zhome request finishes.
Then the old code would never add that selected_message_id to the
zhome table, so it wouldn't show up in the Home view or if you
re-narrowed.
I'm pretty sure there are still cases where the selected_message_id
might be outside the range of messages in zhome, in which case adding
to zhome might put old messages at the bottom. I observed this twice
on staging but was unable to reproduce it consistently.
(imported from commit 162feff3090f8806cc67140db0cfabb6e965aece)
The schema change made this no longer a guarantee, and creating duplicate
PUs is not harmful.
(imported from commit ca05b4dad7d3cc12a35548817f98c0a072c14c8f)
This involved splitting off into a separate function the second step of
the two-step registration process.
(imported from commit bd1ec9b5abb15d29b0585873daaa54a8d4e26556)
Added field invited_at on zephyr.PreregistrationUser, with a one-time
default of Jan 1, 1970 for existing objects.
Added M2M table for streams on zephyr.PreregistrationUser
Deleted unique constraint for ['email'] on zephyr.PreregistrationUser
(imported from commit 85247acb488201f8fc51dfaae354423c27eddcb0)
This allows us to handle the return_messages_immediately part of
get_updates requests without having to talk to the database.
(imported from commit ed0b7742d359efb21a0a4960f4fc25f4337e9ad4)
This commit has the effect of eliminating all of the non-UserActivity
database queries from the Tornado process -- at least in the uncached
case.
This is safe to do, if a bit fragile, since our Tornado code only
accesses these objects (as opposed to their IDs) in a few places that
are all fine with old data, and I don't expect us to add any new ones
soon:
* UserActivity logging, which I plan to move out of Tornado entirely
* Checking whether we're authenticated in our decorators (which could
be simplified -- the actual security check is just whether the
Django session object has a particular field)
* Checking the user realm for whether we should sync to the client
notices about their Zephyr mirror being up to date, which is quite
static and I think we can move out of this code path.
But implementation constraints around mapping the user_ids to
user_profile_ids mean that it makes sense to get the actual objects
for now.
This code is not what I want to do long-term. I expect we'll be able
to clean up the dual User/UserProfile nonsense once we integrate the
upcoming Django 1.5 release, with its support for pluggable User
models, and after that I change, I expect it'll be fairly easy to make
the Tornado code only work with the user ID, not the actual objects.
(imported from commit 82e25b62fd0e3af7c86040600c63a4deec7bec06)
Otherwise one gets:
AttributeError: 'module' object has no attribute 'time'
when trying to use the time module from inside zephyr.lib.
(imported from commit 645368672a3eff68320278dd480edeed56721fcc)
We suspect that these seem to be causing a regression where scrolling
in narrowed views gets really sluggish, but we haven't totally been
able to figure out why since it's challenging to reproduce locally.
(It currently manifests itself on staging but not prod.)
So for now we'll back them out. Here's the full set of things:
Revert "Cause update_floating_recipient_bar to get called less frequently."
This reverts commit a6c1518c4001a2dde44d7b512236795da3ccd351.
Revert "Remove double-scroll in un-narrowing code."
This reverts commit 3dde6c27ffa1e8afa1a084b1b2baee3bc0512962.
Revert "Reset our scroll position if we change our hash to "#"."
This reverts commit 925b44d770c96dafaabebc9e0114f9a3b8f53c4d.
Revert "Properly update floating subject bar when you are at top of page."
This reverts commit 6633cc8a81aedcbb31b30d7c1f27816f8808c700.
(imported from commit a273730581cef30c33bedf701659ee084434f8ad)
Note that on local dev servers, this will print out every half second, as
Tornado polls for file changes for autoreloading. In production it will only
print out when network events occur.
(imported from commit adfe88879e4e446b7dfa6ee69e0a9ad013e9c4d4)
Zev Benjamin
Keegan McAllister
Leo Franchi
Jacob Hurwitz
Waseem Daher
Leo Franchi
Luke Faraone
Tim Abbott
Jessica McKellar
Luke O'Malley
Jeff Arnold
Reid Barton