This contains the various fixes that needed to be made in order to get
accurate statistics.
Most notably, the active_users_between function in the previous
version of zerver/lib/statistics.py was broken for end dates in the
past, because it used the UserActivity table to get its data -- so in
fact it really was querying "users last active between".
This commit isn't super clean, but I figure we're probably better off
having our latest code for historical usage data in git so it doesn't
bitrot and anyone can improve on it.
(imported from commit 24ff2f24a22e5bdc004ea8043d8da12deb97ff2f)
Features:
* Only shows messages in the narrow
* New messages in the narrow will arrive as they are sent
* Works even for streams you're not subscribed to
* Automatically subscribes you to a stream on send
* Doesn't update your pointer
* All searches etc. automatically have the narrow added
(imported from commit 2e12b76849f6ca0f53dda5985dad477a04f7bbac)
This sets up a scheme to validate complex data structures and
give specific error messages for improperly typed parameters.
(imported from commit 33b2f070d993da4ee929119dd41503bd0128c8eb)
* Deal with shorter tweet IDs
(some old tweets don't have a full 18-character ID)
* Allow trailing slash
* Deal with old-style #! syntax
* Deal with links that link to a photo
(imported from commit 008a98c806f3b8dddd9e2f18a8f002af6932766f)
These images at least load now, but that's because Camo redirects
the browser to the origin server, so the only effect is an extra
round-trip time.
(imported from commit 0d6b9c888a5cdfaa9299272d74a085e872dfa434)
In plaintext e-mails these will be simple links.
In HTML e-mails these become a <link> and <img>, which some web mail
clients may inline.
(imported from commit b1242dfd917008a019981eb2224c1c7f5f84739f)
You can't have unread PMs sent by you, so we weren't explicitly
checking this, but when testing locally we often ignore the unread
check. Filter PMs sent by you to reduce confusion when testing
locally.
(imported from commit 0205c4a3ed67790b9d60d4f2b927e4cb9e720bf3)
Since we changed the initial subscription data to include
user_profile_ids rather than emails, we need to preserve that when
adding in events generated during the page load.
(imported from commit 4f4071b8ba30e57c6f64c9e7b54c1cc754e8f010)
This will allow us to substantially decrease the server-side work that
we do to support our Mirroring systems (since the personal mirrors can
request only messages that user sent) and also is what we need to
support a single-stream Zulip widget that we embed in webpages.
(imported from commit 055f2e9a523920719815181f8fdb44d3384e4a34)
This caused problems with our tests suite where we were using a logged
in browser session and actually acting as a different user.
(imported from commit 73b8cb39d5d669e682fbacf2f7e574c228885c2f)
This replaces the AppleDeviceToken table with a generic
PushDeviceToken with a `kind` field to make it easier to add functionality
like per-device/per-stream settings that share code between Android and
iOS devices.
The schema must continue to work on prod with the old table name, so we
add the new table in parallel and can drop the old table once this code
hits prod and any necessary data is copied.
(imported from commit 0209a7013f2850ac6311f23c3d6f92c65ffd19e3)
We have observed additional exceptions being thrown from zulip_finish and we
need to make sure that the handler is disconnected from the queue, or else the
event queue will keep throwing exceptions due to the handler being closed.
(imported from commit 59273aa14495216430b9eb1525b2cce230d8913d)
This should dramatically improve the speed of the dump/load part of
our restart process, especially with large long-lived event queues.
(imported from commit ae4ae20ba2ca4433e25a5e7beeb4fa4882c53972)
Previously, we had an issue with the ACKing protocol, where if a
virtualizable event (like a "read" flag) was dispatched to a queue
client immediately, we would not properly ACK the change because it
had been made a virtual event.
(imported from commit ea09812f8a5ba1d5aad65f536022e3dbc77b0f9e)
Unbundle the push notifications from the missed message queue processors
and handlers. This makes notifications more immediate, and sets things up
for better badge count handling, and possibly per-stream filtering.
(imported from commit 11840301751b0bbcb3a99848ff9868d9023b665b)
This should dramatically improve the speed of the dump/load part of
our restart process, especially with large long-lived event queues.
(imported from commit cc493fa50b4c339257e060b3f0c0956c682e449d)
Now that we support email aliases, we have to be careful when going from
an email address to a domain that we assume we can use to get a Realm
object for. When we care about the Realm's domain, we want to follow
any RealmAliases that exist for a certain domain.
When we just care about the original email address domain itself,
for comparison or other purposes, use split_email_from_domain
This removes the ambiguity of having to decide when to use
email_to_domain + RealmAlias or just email_to_domain
(imported from commit 0e199495502d946ce2e1aae56263e7e8665be4ed)
Now we can nest fenced code/quote blocks inside of quote
blocks down to arbitrary depths. Code blocks are always leafs.
Fenced blocks start with at least three tildes or backticks,
and the clump of punctuation then becomes the terminator for
the block. If the user ends their message without terminators,
all blocks are automatically closed.
When inside a quote block, you can start another fenced block
with any header that doesn't match the end-string of the outer
block. (If you don't want to specify a language, then you
can change the number of backticks/tildes to avoid amiguity.)
Most of the heavy lifting happens in FencedBlockPreprocessor.run().
The parser works by pushing handlers on to a stack and popping
them off when the ends of blocks are encountered. Parents communicate
with their children by passing in a simple Python list of strings
for the child to append to. Handlers also maintain their own
lists for their own content, and when their done() method is called,
they render their data as needed.
The handlers are objects returned by functions, and the handler
functions close on variables push, pop, and processor. The closure
style here makes the handlers pretty tightly coupled to the outer
run() method. If we wanted to move to a class-based style, the
tradeoff would be that the class instances would have to marshall
push/pop/processor etc., but we could test the components more
easily in isolation.
Dealing with blank lines is very fiddly inside of bugdown.
The new functionality here is captured in the test
BugdownTest.test_complexly_nested_quote().
(imported from commit 53886c8de74bdf2bbd3cef8be9de25f05bddb93c)
This reverts commit 1147814b22fb9737a807057ddbdbe0e9554086e0.
This seems to with some probability screw up our Zephyr mirroring
script.
(imported from commit 4f82452f1b0ca98e6b895db020e071d2daa325e4)
This requires a puppet apply on each of staging and prod0 to update
the nginx configuration to support the new URL when it is deployed.
(imported from commit a35a71a563fd1daca0d3ea4ec6874c5719a8564f)
There will be browser errors on staging when this is deployed due to the socket
protocol changing.
(imported from commit f1eda5b5c2ec9c60c23b3ca96277a61debadf5bb)
I believe there may also be others. I'm still not sure why clients would be
sending open requests without session or csrf values in their cookies, though.
(imported from commit 7e9660c1c4d5c2abf55ff21b433ba0117180eb82)
CUSTOMER13 doesn't want it, and there's currently no nginx config
or configurable Camo URI, so it wouldn't work if image preview
were enabled.
(imported from commit 615d4a32acbc4d4d590f88cf4e7d45d8f49db1d3)
ScheduledJobs with type Email displace the usual mandrill codepaths
in the Zulip Enterprise deploys
* Email-specific helper functions will appear in deliver_email.py
* 0058_auto__add_scheduledjob.py
(imported from commit 8db08d8a279600322acfdbed792dc1a676f7a0ab)
The !gravatar markdown no longer hard codes to Gravatar, but
instead it serves up our generic avatar URL.
(imported from commit 4e3e2baeb3374bcf025a18ff27a8452b975c22b7)
We also now separate out the times for the socket overhead, the
request service time, and the queuing delays.
(imported from commit e1683f7f28b968b86ebb701b0ac29b00ac6d67c3)
One quirk here is that the Request object is built in the
message_sender worker, not Tornado. This means that the request time
only counts time taken for the actual sending and does not account
for socket overhead. For this reason, I've left the fake logging in
for now so we can compare the two times.
(imported from commit b0c60a3017527a328cadf11ba68166e59cf23ddf)
This logging is kinda excessive since it adds like 4 log lines per
recipient, so I expect we'll end up reverting it once we've debugged
the proximal issue.
(imported from commit 5e6ab3e230f32b65ad9cf0d95f20ffbc0fe7397e)
This will hopefully help with the send dialog being stuck on
"sending" as well as allowing us to not show errors to the user on
reconnect.
(imported from commit 31ee889853f348e486863073dc130cdfb4e1338d)
Clients can only have one connection at a time, anyway, so we can
just keep track of a client id, instead. This makes reconnections
easier.
It's a little funny to use queue ids for the client id, but we know
they should exist by the time the client is connecting and they are
guaranteed to already be unique and authenticatable. We will also
eventually be integrating the event system and the socket code closer
anyway.
(imported from commit 1f60e06fb16d31d6c121deafd493fb304d19a6c2)
If you don't have a cookie or basic auth and the request looks like
a top-level page in the browser, redirect to the login page.
(imported from commit fc1bcb1080591522bd1b694664255f7049a5d443)
The register_json_consumer() function now expects its callback
function to accept a single argument, which is the payload, as
none of the callbacks cared about channel, method, and properties.
This change breaks down as follows:
* A couple test stubs and subclasses were simplified.
* All the consume() and consume_wrapper() functions in
queue_processors.py were simplified.
* Two callbacks via runtornado.py were simplified. One
of the callbacks was socket.respond_send_message, which
had an additional caller, i.e. not register_json_consumer()
calling back to it, and the caller was simplified not
to pass None for the three removed arguments.
(imported from commit 792316e20be619458dd5036745233f37e6ffcf43)
Looking at the historical data, fewer than 50% of active users have
completed the checklist, which means that it is just persistent
clutter. We also have other better ways of encouraging people to send
traffic and get the apps now.
This commit removes both the frontend UI and backend work but leaves
the db row for now for the historical data.
(imported from commit e8f5780be37bbc75f794fb118e4dd41d8811f2bf)
Here we introduce a new Django app, zilencer. The intent is to not have
this app enabled on LOCALSERVER instances, and for it to grow to include
all the functionality we want to have in our central server that isn't
relevant for local deployments.
Currently we have to modify functions in zerver/* to match; in the
future, it would be cool to have the relevant shared code broken out
into a separate library.
This commit inclues both the migration to create the models as well as a
data migration that (for non-LOCALSERVER) creates a single default
Deployment for zulip.com.
To apply this migration to your system, run:
./manage.py migrate zilencer
(imported from commit 86d5497ac120e03fa7f298a9cc08b192d5939b43)
Trac #1734
This is implemented by bouncing uploaded file links through a view
that checks authentication and redirects to an expiring S3 URL.
This makes file uploads return a domain-relative URI. The client converts
this to an absolute URI when it's in the composebox, then back to relative
when it's submitted to the server.
We need the relative URI because the same message may be viewed across
{staging,www,zephyr}.zulip.com, which have different cookies.
(imported from commit 33acb2abaa3002325f389d5198fb20ee1b30f5fa)
This has a small bug where we don't actually filter the message out of
the home view; fixing that requires adding an index on the "flags"
field of UserMessage.
(imported from commit 492c99d0a8e87b253e577be6564bec12099bd8e9)
Because our authentication system reads cookies from the initial
connection attempt, several SockJS transports can't be used.
(imported from commit 34b9571225d39072985b8223fb12c43c7235841f)
New dependency: sockjs-tornado
One known limitation is that we don't clean up sessions for
non-websockets transports. This is a bug in Tornado so I'm going to
look at upgrading us to the latest version:
https://github.com/mrjoes/sockjs-tornado/issues/47
(imported from commit 31cdb7596dd5ee094ab006c31757db17dca8899b)
Jessica McKellar
Tim Abbott
Steve Howell
Waseem Daher
Kevin Mehall
acrefoot
Leo Franchi
Zev Benjamin
Jessica McKellar
Luke Faraone
Tim Abbott