South doesn't properly deal with removing the Django User model, so
this commit redoes our South history to instead start after that
migration has already been applied. This allows us to get rid of some
annoying hacks.
Note that developers and staging will need to run
./manage.py migrate --delete-ghost-migrations zephyr
in order to clear out the old versions of the migrations.
(imported from commit 7f45ea601b809dde33720f76e7dfb0ab348b0e65)
This, combined with acrefoot's work on sending the notifications in
bulk, resolves trac #1142 -- we do only 10 database queries and the
whole operation completes in about 300ms on my laptop.
(imported from commit 36b5bb836bc6c713903d1ca72e39af87775dc469)
I renamed set_count_internal to update_count_in_dom, because "internal"
was redundant in terms of saying the function was private, and it misled
me into thinking it was internal-only in impact, but it actually updates
the DOM.
I also removed the synchronous callback functions, since they both
led to simply hiding the count_span and clearing the text of the
value_span.
(imported from commit dea27d6414dc1b33818b24662f8246d687530b71)
This allows us to load our own code before most dependencies are
loaded. Our compiled Handlebars files still need the Handlebars
runtime, so we can't move all of our minified code before
dependencies yet.
(imported from commit e2d0fa13f05a08fc3c2519790f7382e5eef6eca2)
The problem is that if you load a browser window in a stream narrow,
add_message_metadata will be called for the messages in the narrowed
view before it is called for the messages going into the main view
(thus inserting them into all_msg_list), resulting in duplicate
copies of messages.
This would be mostly OK except that we call
process_message_for_recent_subjects inside add_message_metadata, and
that function assumes it is only called once on each message
(otherwise it'll double-count the message).
(imported from commit a3e7f85874100cd93a6d07684605da04d9cc80c7)
Created a function message_viewport_info() to return more accurate
effective viewport info and called it from process_visible_unread_messages().
Also killed off a tiny bit of dead code in process_visible_unread_messages().
(imported from commit 985fcf2fb447dbf1026e2de37574c255a9bd6196)
Whenever we get a narrowing event, it's possible for new messages
to appear visible, and we need to call process_visible_unread_messages().
This has been a bug, but it's mostly obscured by the fact that we
call process_visible_unread_messages() as part of focus/scrolling
events.
(imported from commit b9447977f8e2272d45865ca67b436cacafd58a03)
This does the simple thing, which will work as long as the phrase
doesn't have any punctuation and contains the exact text being
searched for. We really want phrase search (check if the lexemes in
the query occur in sequence in the input lexeme string), but Postgres
doesn't support that natively.
(imported from commit 67bf36883ed21743fcad3f02ad5b319ab188f816)
This works fine on prod, but after a new install-server, the closure
compiler complains about a side-effect-free for loop init.
(imported from commit aa0e4d788abe4c819d4d912d6a526fab4f676675)
This removes the large "New stream message" and "New private message"
from the left sidebar. It also makes the default action when clicking
inside the composebox the same as the "New stream message" button used to
do (instead of replying to the stream-subject pair at the current cursor).
(imported from commit 316f03a35b781aca4c42555f74b99c4332ff42de)
THE CENSUS OF HUMBUG
Now it happened that at this time Waseem Daher issued a decree that
a census should be made of the whole users of Humbug.
This census -- the first -- took place while Faraone was governor of
MailChimp, and everyone went to be registered, each to his own realm.
So Alice Humbugger set out from the town of MIT for MailChimp in order to
be registered.
(imported from commit fca7714ebffd0b39b9b1337058f67975985f4039)
This refactoring basically splits off two functions from update_unread_counts(),
which then becomes a simple three-liner.
The function get_unread_counts() is extracted, and it's purely functional
computation. It paves the way for a more pull-based approach to getting "unread"
counts, where other parts of the program can just call it to get values as
needed without worrying about side effects. It is staying in zephyr.js for
now.
The other function is stream_list.update_dom_with_unread_counts(), which
has a new home in stream_list.js. It handles all the DOM manipulation
aspect of unread counts in the left pane, mostly by delegating to smaller
functions within stream_list. Some of those smaller functions can now
be turned into private methods FWIW, but I'm not sure it's worth the
trouble.
(imported from commit 799f9ebbaed8d530829a4741ef14be04bd8abf5a)
This is a prefactoring to eventually eliminate the home_unread_messages
global variable. More commits to follow.
In order to set up process_loaded_for_unread() not to modify
global variable to get its job done, we want to pull it out of
add_messages(), so that add_messages() doesn't have to pass back
state to the 9 different places in the codebase where it's called.
There are only 2 places where process_loaded_for_unread() get
called after this commit.
In order to facilitate pulling up process_loaded_for_unread(), I
made it so that the contract for add_messages() was to accept
already-hydrated messages. This way I could hydrate the messages
before calling process_loaded_for_unread() without have to
worry about double-caching them in add_messages. This will
slightly improve performance, but it was mostly done for code
clarity.
(imported from commit ad5aaad5b1f22c31647370f4c9dcb5f89d7d99a7)
This was a workaround for a number of other bugs we had, but at this
point just serves to make debugging more difficult.
(imported from commit 6662b7854c265bd8016f6c8ce75a095731211a45)
Since we log to statsd our cache time lookups by cache key, using a unique
tweet id for each lookup was just filling up our cache without being useful.
Also, log database cache lookups in a further namespace to distinguish between
memcached caches
(imported from commit a2a16b777fb7ab8cd066feee7344f9c8a3c107f5)
Users can send to any stream except invite-only streams that they
aren't subscribed to. Bots can send to any stream except invite-only
streams that neither they nor their owner is subscribed to.
(imported from commit 623d34d249d923611ca7ca781b5b55205cd3e548)
We really should not be storing bot API keys in the DOM and should
require some sort of additional authentication before showing them,
but this seems reasonable for a first pass.
(imported from commit c7d75aa52e21894bf53917457e771c18de38bbcc)
Previously a default was missing from this function, which resulted in
users being unable to change their settings if they tried to disable
sounds.
(imported from commit 2dae67dcb2e8cb986abb6dee9659be2192993dd9)
After this change, the memcached time consumed by doing
get_old_messages for 200 and 1000 messages respectively now look like
this:
200 63ms (mem: 6ms/3) (db: 4ms/2q) /json/get_old_messages
200 178ms (mem: 67ms/2) (db: 6ms/1q) /json/get_old_messages
which might help explain where the time is going on prod for some of
our slower queries.
(imported from commit b8fe83b175914b6796922a65a1c5537f4e7a9429)
The idea here being: if there's only one line, it discourages
me from writing a long message (and also makes me think that
enter will send).
(imported from commit 424d8d305d1965ce3199ce3227dac94b395945bc)
This commit takes control of keyboard-based pagination away
from the browser, so that we can use the effective viewport
size as the amount to page, as well as keeping a little bit
of overlap from page to page. There had been issues with
pagination for a while, but the introduction of the always-open
composebox particularly aggravated the situation.
(imported from commit 45b9b7d5a6b322230c9d55e1be0b763dbce06e2e)
For sites that are supported, we now grab thumbnails for images + video
embed code for videos and use them in lieu of our existing embed code.
We also embed rich non-script content.
Special casing is done so that we don't embed images twice.
Some testcases were modified to avoid triggering Embed.ly
The manual step is to install python-embedly.
(imported from commit d725bab91675c61953116c5ca741055fce49724e)
Previously we never sent desktop notifications when the browser was
focused, even if the message appeared offscreen. After this commit
there are only a few cases when PM or other notifiable message doesn't
trigger a desktop notification:
(1) You sent it yourself
(2) It was onscreen when it arrived while your Humbug window had focus
(imported from commit e381c02c0e6794594d6934f57249a11ba2a88210)
This is trying to make the logic flow clear -- e.g. we check once, at
the beginning, for whether the message is notifiable, and the checks
for whether the various notification settings are enabled are more
parallel.
(imported from commit a68c71a53055191bc16682a85f739ed8e40ddeae)
It's strictly more functional, and having a single arguments
extraction decorator makes our codebase less confusing.
(imported from commit 2a5618c04b486268a462a24a1481ac030f15eac4)
See #1234 for details. When you upload files the old-school way
(no drag&drop), there was a bug where you couldn't upload the same
file twice, due to us intercepting the change event and not clearing
out the file list when we were done. Tested on Chrome, but uses
a known IE workaround.
(imported from commit 8120c2e8bce41f3964f4f5c21aad3a85df0e433d)
The filedrop library has a few canned errors, but it seems to mostly
let server errors come through. We try to trap 413s to give a more
descriptive error than "unknown," but this is just a bandaid fix,
and we should see what's wrong with our prod configuration.
(imported from commit eac26406866d80340f24dbdca9f34408ddb92462)
The .height() and .width() functions are actually pretty expensive for
the number of times we call them. The viewport height and width
don't change often, though, so we can just cache them and recalculate
them on window resize.
(imported from commit 129fb8c058144125e2974f6b7967cd9f1a5c9ead)
The .height() and .width() functions are actually pretty expensive
for the number of times we call them. The callers of within_viewport
already know the offset and height of the row, so we just pass them
in so the values don't have to be recalculated.
(imported from commit d1c077bd87463d695f0bbe337b6a8b04ac2d17ce)
The optimizations are:
* Sort over the list of subscriptions instead of the DOM li elements.
This requires storing the li elements for each sub on the sub object.
* Do a bulk insert of the li elements instead of doing them one by one.
(imported from commit 1a987799930fc677e25f0bc2dcf66f83a4ac3163)
We now fire three events:
* subscription_add_done - fired when subs.js has finished handling a
subscription_add event (all structures are set up, etc.)
* subscription_remove_done - fired when subs.js has finished handling a
subscription_remove event
* sub_obj_created - fired when subs.js has created a sub object. This
happens both when a new subscription is added and at page startup for
all existing subscriptions
These events are fired whenever sub objects are created, even when
not tied to a subscription event.
(imported from commit a4863451f37e7fdbad480696b388ea788b01d6b9)
* Start a compose when we do a file upload
* Restore the "Formatting" and "Feedback" links.
* Dismiss composebox error messages when we defocus composebox
Realistically, the "correct" way to do this is not to have to
explicitly manage the composebox's state, as we do now -- it should
just be 100% visible and ready to send any time you click 'send'; it
shouldn't need to have first been composebox.start()ed.
(imported from commit 7f1725c229ed968a9b5500b25d600306173182a0)
What changed:
* Vector icons swapped in for the left sidebar buttons and filters
* Lighter font weight in the stream filters list
* Round color swatches in the stream filters list, with an inner shadow
* Tighter line height in the individual messages in the message pane
* Fixed button widths in the left sidebar (so the buttons are equal in width)
(imported from commit 337dc4a3d8e29945cfc8cfb9524ac76a7b038ad8)
Recently the typeahead for streams in the compose box was modified so
that streams only matched queries when the query was a prefix to the
stream. When that change was made, the old highlighting behavior
had been mistakenly left in place. This commit fixes the highlighting.
(imported from commit b7ec33daba46978df58eb91306686a4f1a57c7fa)
* Properly resize compose area when we cancel out of it.
* Re-enable clicking on 'reply' in popover.
(The issue with the latter is that clicking on "Reply" started
a reply and then bubbled up and triggered our code that canceled
a reply because you clicked out of the composebox.)
(imported from commit 25d0ea58b72d2ee246217baf3eb9cac58fc858f5)
Really, the "correct" way to do this is to undo "scrolltheworld", and
then just have a compose div that always lives underneath the message
list div. (This will also allow us to deal much more reasonably with
the whole "Is the composebox in focus" thing.)
In the interest of prototyping something more rapidly, though, we
adopt the somewhat more hackish approach, with the understanding that
much of it will probably be simplified later.
(imported from commit e2754be155c522b6dac28e7b84c62bd2030217c8)
We previously kept the lists in the DOM for all streams and updated
them all when new messages arrived. This was very expensive for
large numbers of streams, so we now just build the subject lists on
demand.
(imported from commit 937ad4322ce2014200aeae8645f79875f6af576e)
This commit also fixes a bug where "starred messages" wouldn't get
bolded when you narrowed to starred messages. However, it also
introduces a regression where subjects aren't highlighted correctly
on load to a narrow which will be fixed shortly.
(imported from commit 411575d92762e41d04c1baf126c0ab1dfb4225a5)
This will matter shortly as hashchange.initialize can call
narrow.activate(), which fires an event handler.
Really, I have no idea why we have these initialize() methods anyway
and we don't just do initialization on document.ready.
(imported from commit 3a6a80e1426b03439b95cae3f142a4b1c43125e9)
We memoize add_message_metadata by checking if the message is already
in the all_msg_list. Therefore, we need to add messages to that
message list before we add it to the narrowed_msg_list.
(imported from commit 4346179376ef6f982162c02c6152a0d294bfb2c0)
The String.localeCompare function is really slow, at least partially
because it creates a locale-aware collator object each time. So now,
when we can, we create and cache a locale-aware collator object.
However, this is not supported on most browsers, so we fall back to a
non-locale-aware comparison. This is not ideal, but for now we are
mostly working with English-speaking customers.
(imported from commit 51aa02e3b9fe4a0ef0cb084874fe26e91c57f65e)
Addionally, print out a blueslip error instead of dying
if a stream id is accessed when there is no stream to get
(imported from commit 0d6466ca79312a4fb9a235f313303ac5246afb35)
This decouples from Chrome notifications, which gives us cross-platform
support in at least modern browsers.
We log this action so its replayable in our message logs.
This implements the model change indicated by the previous schema commit.
(imported from commit b21213cdde54f43670bbb0bf1f607147fc732b38)
We test if the user supports sound in their browser, then determine which
sort of sound their browser supports.
When, whenever we show a desktop notification we also play a sound.
(imported from commit dae41e70a6e4f6ed60ffedaac546d77baee52675)
Since they can't be parsed, probably the best thing to do is to send
the user to the home tab; we could add in showing an error message but
then we'd need a way to clear the error message -- better to just have
this work.
(imported from commit 67c0475ff06eb0431621eef60b9c50287a158232)
Previously, we were fetching Message.objects.select_related() from the
database, even if we actually ended up fetching the message dicts from
memcached and thus not actually using them. Especially in the cached
case, this resulted in a lot of overhead where the Django ORM put
together Message objects with lots of data in them that were never
used. This commit switches the model to only fetch the full message
objects from the database for those messages which are not found in
the memcached caches.
Here are the timings for get_old_messages before this patch was applied:
(cached)
127ms (db: 42ms/2q) /json/get_old_messages (starnine@mit.edu via website)
385ms (db: 105ms/1q) /json/get_old_messages (starnine@mit.edu via website)
(uncached)
315ms (mem: 6ms/41) (db: 90ms/22q) /json/get_old_messages (starnine@mit.edu via website)
507ms (db: 94ms/14q) /json/get_old_messages (starnine@mit.edu via website)
Here are the timings for get_old_messages after this patch was applied:
(cached)
80ms (db: 9ms/2q) /json/get_old_messages (starnine@mit.edu via website)
133ms (db: 4ms/1q) /json/get_old_messages (starnine@mit.edu via website)
(uncached)
230ms (mem: 9ms/41) (db: 48ms/23q) /json/get_old_messages (starnine@mit.edu via website)
385ms (db: 55ms/15q) /json/get_old_messages (starnine@mit.edu via website)
(imported from commit c4748513392a906393314aa7cd41d98a69865411)
The .data() method tries to coerce the value of the attribute into a
Javascript type, which is not what we want when the stream name looks
like a number or some other Javascript type.
(imported from commit a5f639d2ef98435cec6beacf3837fc185474a955)
On page load, the scroll_finished function was being called and
scroll_start_message was -1. This caused us to mark all messages
that we loaded through the messages initially visible as read. This
was particularly problematic because message_range iterates over all
message ids between its two arguments.
(imported from commit d93209d466797939cc9dbdbe76d25a5b20195bd2)
Previously we were doing quadratic work in the number of streams
because we had to iterate over all <li> elements every time we added
a new one.
(imported from commit 60cb97f77d161e9d8c3072157fa9c57c58f7af52)
Since we pick a new color every time we add a new subscription and
recomputing the available colors was linear in the number of
subscriptions, we were doing quadratic work on page load.
(imported from commit 647ff3cb82f405755711da47701f005e7bc0023e)
We were previously doing this on every message. Because
update_recent_subjects is linear in the number of streams in the
sidebar, this became very slow when we enabled the streams sidebar
for the MIT realm.
(imported from commit 95cd71d83bbcc08cc6c5c79ca567b5d6b9b17173)
We were previously calling sort_narrow_list after each stream was was
added. Because it is linear in the current length of the sidebar
list, we were doing quadratic work on page load. When we enabled the
streams sidebar on the MIT realm, this became problematic because of
the number of subscriptions Zephyr users have.
(imported from commit d60ddc638f0a81fbce08eecd6671e9ea6ca38515)
Messages are now explicitly condensed by our JS, which means that if
we run into some bug where our JS doesn't run, you still see the whole
message (rather than getting a clipped message).
(As of this commit, this can happen when you, e.g. are on the
Settings page and someone sends you a message.)
(imported from commit f3bec97800ea1852c80203e73552ee545fcc7e8a)
This fixes a bug where if you were narrowed to a search and received
a new message that belonged in that search, the message would appear
to have an empty subject and content.
(imported from commit fe1dbf584d3659d57c5b70c7eb45cb22bbc9732f)
Previously, we were having this problem where:
* You narrow to something
* That causes message_list.js:process_collapsing to run on all of the
elements in the view, which changes some of their sizes
* That causes the pane to scroll and either push the content up or
down, depending (since stuff on top of where you were is now a
different size)
* That triggers keep_pointer_in_view, which moves your pointer
Moving process_collapsing into narrow.activate doesn't obviously
fix any of this, but it does seem to mitigate the issue a bit.
In particular, we (a) process it less frequently, and (b) process it
immediately after we show the narrowed view table, which seems to
reduce the raciness of the overall experience.
This does, however, introduce a regression:
* If you receive a long message when you're on
#settings, e.g., and then go back to Home,
the message does not properly get a [More] appended
to it.
(imported from commit b1440d656cc7b71eca8af736f2f7b3aa7e0cca14)
This can be useful for debugging what sort of narrow is happening in
addition to the URI decoding bug we're currently experiencing.
(imported from commit 0cb55fec4ac1afa986c747eb79236b4300c9e636)
This shouldn't have any effect in normal realms, but for realms like
mit.edu that have large numbers of inactive streams, it will sort all
the streams that have had a recent message at the top (aka those that
aren't effectively inactive).
(imported from commit 027ce258d04b6fd58705e49f769dec7e0639bb38)
We HTML-escape the subject in Postgres to avoid a server round-trip.
Unlike the rendered_content, which is already escaped and cached on
zephyr_message, we normally escape subjects client-side. Escaping in
Django would require fetching the messages that match the query,
escaping the subjects, and then making a second query to Postgres to
insert the markup. We could instead fetch the messages with subjects
marked up using non-HTML (some unique string) that is later converted
into the correct markup either in Django or client-side, but then the
escaping problem would just be with some random string instead of
HTML. Since the function is pretty simple, doing the escaping in
Postgres itself is the least painful option.
(imported from commit 004931d8e496697c18650aee97b1a74c55a04cb2)
Tim Abbott
Steve Howell
Zev Benjamin
Luke Faraone
acrefoot
Leo Franchi
Allen Rabinovich
Jessica McKellar
Waseem Daher
Jeff Arnold