Zendesk works a lot like desk.com, it has triggers which use targets.
The triggers have a user defined template. Targets can also have place
holders that are posted, we add the ticket id and title here so we can
always construct the message subject.
(imported from commit 04e8e5c7c0fc5568201f252546f6ed42f282fd00)
The normal code will now generate SQL that is basically identically to the raw
SQL we were using before.
(imported from commit 84a3971d6137d05ef3f71252278afdd59041e86a)
This commit also includes a few changes in the way we do some queries that
should speed searches up:
* Messages before and after the anchor are fetched in a single query by doing a
UNION ALL on the server. This incurs the cost of an extra sort because UNION
ALL does not guarantee the order that the results are appended, but it saves a
server round-trip.
* Searches involving flags now use a straight froward WHERE clause, which is
much faster than the one that django-bitfield generates (due to limitations of
the Django ORM)
(imported from commit a0db811a9073363cfabcf4b035d02d20dc8fc8a4)
This requires the tsearch_extras Postgres extension. To install the extension,
first install postgresql-9.1-tsearch-extras on both postgres-primary and
postgres-secondary (this would normally be done in a puppet apply, but there are
currently some changes that can't be applied on Postgres machines). Then run
the following as the postgres user on postgres-primary:
$ psql -d zulip -c 'CREATE EXTENSION tsearch_extras SCHEMA zulip;'
In dev environments, you must also run:
$ psql -d zulip_test_template -c 'CREATE EXTENSION tsearch_extras SCHEMA zulip;'
(imported from commit ad0a57c455b3b86002191ac5fb705d8f716f3296)
This returns us to the original situation where we're not in the
practice of doing database fetches for UserProfile objects in the
Tornado workflow, which should make our performance no longer
alarmingly degrade when memcached gets flushed.
(imported from commit 3580737604c13c9989f120a2242ad4c3eab6e1d1)
This does result in a few more rabbitmq events to be processed (though
a negligible number compared to what we already do), but it saves a
database query from inside Tornado whenever we occasionally have a
cache miss looking up the UserProfile, which is far more important.
(imported from commit a553a00a3004ba27bfb54ffbc3e9c9b170ebae4d)
This is used by the Android app to authenticate without prompting for a
password.
To do so, we implement a custom authentication backend that validates
the ID token provided by Google and then tries to see if we have a
corresponding UserProfile on file for them.
If the attestation is valid but the user is unregistered, we return that
fact by modifying a dictionary passed in as a parameter. We then return
the appropriate error message via the API.
This commit adds a dependency on the "googleapi" module. On Debian-based
systems with the Zulip APT repository:
sudo apt-get install python-googleapi
For OS X and other platforms:
pip install googleapi
(imported from commit dbda4e657e5228f081c39af95f956bd32dd20139)
Previously we unconditionally showed the "get the desktop app"
banner. Now, if the first user declines to invite people as part of
their onboarding workflow, show the invite banner instead.
(imported from commit f7892fef17c923154a700149b8f5be99e9c03fa0)
We currently only do bulk invites when the first user in the realm
goes through the signup process, so this will help us know if that
step is effective for getting more early users into the app.
(imported from commit c846086185ed28b13d3d4b695a9c8cad913d3bc9)
When we edit a message, send out UserMessage flags to the recipients.
This sets the stage for making sure that changes related to
user-specific alert words or mentions get sent out to users.
(imported from commit bce1de19acef44b5e106352f261203352ece02b9)
Image and video links in the twitter API are media and need to be
handed on separately. We also include a preview image if the media link
is a to a picture.
(imported from commit 2bd00d267e51b29ad0ba681195b2bfea9b991d8c)
This converts links in tweets to a tags. We also convert the displayed
text to the target of the twitter short URL. Mentions are linked to the
users twitter page.
(imported from commit 192d5546a7eea82759f9ae30d82c102aed15ff71)
Previously, we were processing in Tornado mirroring dummy users (and
deactived users) as recipients -- resulting in bugs where the missed
message hooks would fire for these nonexistent users.
Our Tornado real-time delivery system only needs the list of active
users both for delivery and for presence information updates.
(imported from commit b81143f106a4d0eefa4b838e7c074b2963259746)
Before this is deployed to prod, we need to manually frob our database
to set the is_mirror_dummy=True bit for all existing mirror users.
(imported from commit 39f1938cef091cf1d7d97307f76b137fe1d92b6c)
NarrowBuilder.by_stream and NarrowBuilder.by_topic for mit users uses a
regex to search by stream and topic. Python's re.escape escapes unicode
in a format that postgres can not parse. We escape unicode as '\uXXXX'
for postgres.
(imported from commit d2c27d4514c31fdc6ef1fea898fe721a6f0ab069)
Avoid repeating request.client = get_client(foo) 5 times, and
decouple the logic for determining the client name from how
we represent the client as an ORM object and piggyback it on the
Django request.
(imported from commit efdf81cbaf9599a5606da18e06d7ffe9b88aaf6f)
The "else" branch used to execute when it was not the case
case that HTTP_USER_AGENT was not in request.META. Now
it's more readable.
(imported from commit ea0664e89d8b3d94cd1c0f8667047461b149ba41)
Two tests were added.
1) That that name of the stream is changed.
2) That realm admin is required.
(imported from commit f8fd482c653c983182b96d53c30d731e272f96cd)
deactivate_stream_backend was untested now we test
1) That subscriptions are removed
2) That the realm admin is required
(imported from commit eedb1c4fc0e363df58721302e9f8fbedf78389a9)
expected_messages was ltriming headings but we had a newline at the end
innerText. The ltrim was changed to a trim to correct this.
(imported from commit 5e411c5fc46a2cb675c1268041e95bbb2522c8f9)
We already have a try-except earlier in the file about email_gateway_user, so we don't
need to check for it again.
(imported from commit 2d9fa357fab2605916c5c5cb61961c0a121b1211)
This is the UI piece that finishes the features to let admins
make streams private or public.
(imported from commit 1a193165a6304dc358982e9850a75965fb3a03fd)
After deploying to both staging and prod, double check the docs
are correct here. This fixes the API docs on prod, which had
"POST /api/v1/messages", despite "/api" not being part of the
prod path. Prod docs are here:
https://zulip.com/api/endpoints/
(imported from commit a2c4d316128f88171f4a76074314be64d9bc9728)
While it does work, it's more an accident than intentional behavior
and not something we want to be encouraging (and it's messier code).
(imported from commit 3797147fc21836135a6304412bd3f958873a0576)
This shows the number of messages sent by humans for the last
eight 24-hour periods, for each realm. "Messages sent" isn't a
perfect metric of activity, but it's easier to query with our
current data model than certain other statistics.
(imported from commit 9de3c479640a0b9dbc017b245dda21d951f4efa4)
This contains the various fixes that needed to be made in order to get
accurate statistics.
Most notably, the active_users_between function in the previous
version of zerver/lib/statistics.py was broken for end dates in the
past, because it used the UserActivity table to get its data -- so in
fact it really was querying "users last active between".
This commit isn't super clean, but I figure we're probably better off
having our latest code for historical usage data in git so it doesn't
bitrot and anyone can improve on it.
(imported from commit 24ff2f24a22e5bdc004ea8043d8da12deb97ff2f)
When we rebuild the user list from scratch, set the unread
counts in the templates to avoid multiple DOM updates.
(imported from commit 2d0c9b0fb99b382332e464ba7c3caad95e05363e)
That way if all you do is briefly check Zulip because you got the
email, we'll send you another one tomorrow.
(imported from commit fcbbd264c5e5fea7352f0fee6989e000af7b7bed)
Features:
* Only shows messages in the narrow
* New messages in the narrow will arrive as they are sent
* Works even for streams you're not subscribed to
* Automatically subscribes you to a stream on send
* Doesn't update your pointer
* All searches etc. automatically have the narrow added
(imported from commit 2e12b76849f6ca0f53dda5985dad477a04f7bbac)
Validators are similar to converters, but they don't have
to parse JSON, and they are told the name of the request
variable to help format error messages.
(imported from commit 3c33e301892519c67e70675006d5686d9f013353)
Make sure that principles is a list a of strings (unless it
is None). This includes a unit test.
(imported from commit c2e3f1c0cafc207ceca67d5a174ef4e29a32c6ca)
This sets up a scheme to validate complex data structures and
give specific error messages for improperly typed parameters.
(imported from commit 33b2f070d993da4ee929119dd41503bd0128c8eb)
* Deal with shorter tweet IDs
(some old tweets don't have a full 18-character ID)
* Allow trailing slash
* Deal with old-style #! syntax
* Deal with links that link to a photo
(imported from commit 008a98c806f3b8dddd9e2f18a8f002af6932766f)
These images at least load now, but that's because Camo redirects
the browser to the origin server, so the only effect is an extra
round-trip time.
(imported from commit 0d6b9c888a5cdfaa9299272d74a085e872dfa434)
In plaintext e-mails these will be simple links.
In HTML e-mails these become a <link> and <img>, which some web mail
clients may inline.
(imported from commit b1242dfd917008a019981eb2224c1c7f5f84739f)
You can't have unread PMs sent by you, so we weren't explicitly
checking this, but when testing locally we often ignore the unread
check. Filter PMs sent by you to reduce confusion when testing
locally.
(imported from commit 0205c4a3ed67790b9d60d4f2b927e4cb9e720bf3)
All of our other API arguments use "_" as the delimiter, so we're
migrating this to do that as well (except for the legacy send_message
usage, which we're just hoping will eventually shrink to nothing).
(imported from commit 40964cc08e0f94ba439a61e4f68ed500f74a554f)
This is for the CUSTOMER28 folks, so that they can turn Zulip into a more "chat client" thing.
(imported from commit 373a8afae4998fce5560e7b2bd13804c8fbb39fc)
Since we changed the initial subscription data to include
user_profile_ids rather than emails, we need to preserve that when
adding in events generated during the page load.
(imported from commit 4f4071b8ba30e57c6f64c9e7b54c1cc754e8f010)
This will allow us to substantially decrease the server-side work that
we do to support our Mirroring systems (since the personal mirrors can
request only messages that user sent) and also is what we need to
support a single-stream Zulip widget that we embed in webpages.
(imported from commit 055f2e9a523920719815181f8fdb44d3384e4a34)
This adds coverage on all of our handlebars templates. It renders
each template with representative data and generally performs at
least one sanity check on the DOM that gets created.
Also, the tests output the rendered HTML to .test-js-with-node.html,
which can serve as a way of documenting all of our templates.
(imported from commit 63dd7502457a8199dd35277fc5ab80cd53e2af22)
This caused problems with our tests suite where we were using a logged
in browser session and actually acting as a different user.
(imported from commit 73b8cb39d5d669e682fbacf2f7e574c228885c2f)
We don't use it yet, but the plan is the migrate there and it's better
to just have the filtering in place.
(imported from commit d0e7f40e8a439b8e8751da954e79b5f67226e5a9)
As far as I can tell, we don't actually use this value, but better to
have it be clear.
(imported from commit 3655b87f28b0554ee3db0acb2c0d59543dd093a1)
This replaces the AppleDeviceToken table with a generic
PushDeviceToken with a `kind` field to make it easier to add functionality
like per-device/per-stream settings that share code between Android and
iOS devices.
The schema must continue to work on prod with the old table name, so we
add the new table in parallel and can drop the old table once this code
hits prod and any necessary data is copied.
(imported from commit 0209a7013f2850ac6311f23c3d6f92c65ffd19e3)
This must be run manually on staging after deployment. Once it has been run,
it can be deleted. It only needs to be run on staging, not prod.
(imported from commit 79252c23ba8cda93500a18aa7b02575f406dd379)
This allows us to avoid a circular import when importing models.py
from inside bugdown for the realm-filters-from-database branch.
(imported from commit 7de85b54243132ade6818b080abdc8c5e8ad84f5)
We have observed additional exceptions being thrown from zulip_finish and we
need to make sure that the handler is disconnected from the queue, or else the
event queue will keep throwing exceptions due to the handler being closed.
(imported from commit 59273aa14495216430b9eb1525b2cce230d8913d)
This prevents us from failing if the first or last name is unset.
We fall back to None, which will allow the user to set their name even
if real names are restricted, which is probably better than forcing them
to have no name.
Closes trac #2118.
(imported from commit 1ff8a55022f3a3baf67575b593a679e21c0f3194)
And in the meanwhile, comment what's going on so that we don't break
this in refactoring again later.
(imported from commit a3119cd1eab3d54cb1883f2c8cad0d147cb04ba7)
Currently all of our realms we intend to create are created manually,
and regardless do_create_realm is the correct way to create a realm.
(imported from commit 42280aff461aa17ffee22ab1c7b7f43757648eec)
I'd also like to add a database table to actually store the values
that we get out of this and our send message requests for future
inspection, but for now, grepping logs+statsd is good enough.
(imported from commit 99ef179651850217fe6e82c5e928d122ca91101e)
We convert sender:me to sender:steve@zulip.com at parsing time,
so users will see the canonicalization in the search bar. Likewise
for pm-with.
(imported from commit aa9951f13d4633cfef85f03e5486d607fdef414f)
We now have 2 variablse:
EXTERNAL_API_PATH: e.g. staging.zulip.com/api
EXTERNAL_API_URI: e.g. https://staging.zulip.com/api
The former is primarily needed for certain integrations.
(imported from commit 3878b99a4d835c5fcc2a2c6001bc7eeeaf4c9363)
Now that we've debugged the memory leak, I don't think we need this
anymore.
This reverts commit 1bdc7ee2f72bdebb1cdc94601247834a434614d6.
Conflicts:
puppet/zulip/files/cron.d/rabbitmq-numconsumers
puppet/zulip/files/supervisor/conf.d/zulip.conf
(imported from commit ff87f2aebcbc71013fa7a05aedb24e2dcad82ae6)
This should dramatically improve the speed of the dump/load part of
our restart process, especially with large long-lived event queues.
(imported from commit ae4ae20ba2ca4433e25a5e7beeb4fa4882c53972)
Previously, we had an issue with the ACKing protocol, where if a
virtualizable event (like a "read" flag) was dispatched to a queue
client immediately, we would not properly ACK the change because it
had been made a virtual event.
(imported from commit ea09812f8a5ba1d5aad65f536022e3dbc77b0f9e)
Unbundle the push notifications from the missed message queue processors
and handlers. This makes notifications more immediate, and sets things up
for better badge count handling, and possibly per-stream filtering.
(imported from commit 11840301751b0bbcb3a99848ff9868d9023b665b)
This requires the nodes.js version of jquery. Talk to Luke if you are having issues
installing it.
(imported from commit 86dc91a9a41b2ef9c2dbcc4fb0085109250d7af7)
The node tests can write to .test-js-with-node.html now, so that
when you are unit-testing template-related code, you can use
the browser to inspect the output.
(imported from commit 3cbd7470ec0da1973124f79acb665c3a3e17c580)
R means "I want to send a PM, you can guess the destination"
r means "I want send a stream message, you can guess the destination"
C means "I want to send a PM and specify the destination"
c means "I want to send a stream message and specify the destination"
(imported from commit 755c92aed79ab79089b2e35d2c100582f012736a)
This should dramatically improve the speed of the dump/load part of
our restart process, especially with large long-lived event queues.
(imported from commit cc493fa50b4c339257e060b3f0c0956c682e449d)
Now that we support email aliases, we have to be careful when going from
an email address to a domain that we assume we can use to get a Realm
object for. When we care about the Realm's domain, we want to follow
any RealmAliases that exist for a certain domain.
When we just care about the original email address domain itself,
for comparison or other purposes, use split_email_from_domain
This removes the ambiguity of having to decide when to use
email_to_domain + RealmAlias or just email_to_domain
(imported from commit 0e199495502d946ce2e1aae56263e7e8665be4ed)
Until we can add a banner to help users subscribe, it may be confusing to
narrow to a stream where you are not subscribed.
Partial revert of 390bdef
(imported from commit ea75fc59b979589b975465a3fecffea0f014fcf6)
It's a little weird that these still open in a new tab, but it might
be best to keep them consistent with all other links?
This is a first pass on Trac #1927.
(imported from commit 390bdef790a83af4240ad5f5a82e572ef5824756)
If authoritative data is available from say the LDAP database, we now
ignore the POSTed user name, and don't offer it as a form field.
We fall back to giving the user a text field if they aren't in LDAP.
If users do not have any form fields to fill out, we simply bring them
to the app without the registration page, logging them in using a dummy
backend.
(imported from commit 6bee87430ba46ff753ea3408251e8a80c45c713f)
The latter doesn't depend on the former; we can still fill in your full
name even if you didn't authenticate via LDAP.
This commit requires django_auth_ldap to be installed. On Debian
systems, you can do so via APT:
sudo apt-get install python-django-auth-ldap
On OS X, use your favourite package manager. For pip, I believe this
will work:
pip install django_auth_ldap
django_auth_ldap depends on the "ldap" Python package, which should be
installed automatically on your system.
(imported from commit 43967754285990b06b5a920abe95b8bce44e2053)
This fixes a small memory leak in our queue workers, where we don't
reset the accumulated times contained in our query logging data.
Longer-term, we may want to make something mergable for mainline where
we only store on the connection object the totals; that would be a
fixed amount of emmory per connection and thus not have this problem.
(imported from commit 914fa13acfb576f73c5f35e0f64c2f4d8a56b111)
Now we can nest fenced code/quote blocks inside of quote
blocks down to arbitrary depths. Code blocks are always leafs.
Fenced blocks start with at least three tildes or backticks,
and the clump of punctuation then becomes the terminator for
the block. If the user ends their message without terminators,
all blocks are automatically closed.
When inside a quote block, you can start another fenced block
with any header that doesn't match the end-string of the outer
block. (If you don't want to specify a language, then you
can change the number of backticks/tildes to avoid amiguity.)
Most of the heavy lifting happens in FencedBlockPreprocessor.run().
The parser works by pushing handlers on to a stack and popping
them off when the ends of blocks are encountered. Parents communicate
with their children by passing in a simple Python list of strings
for the child to append to. Handlers also maintain their own
lists for their own content, and when their done() method is called,
they render their data as needed.
The handlers are objects returned by functions, and the handler
functions close on variables push, pop, and processor. The closure
style here makes the handlers pretty tightly coupled to the outer
run() method. If we wanted to move to a class-based style, the
tradeoff would be that the class instances would have to marshall
push/pop/processor etc., but we could test the components more
easily in isolation.
Dealing with blank lines is very fiddly inside of bugdown.
The new functionality here is captured in the test
BugdownTest.test_complexly_nested_quote().
(imported from commit 53886c8de74bdf2bbd3cef8be9de25f05bddb93c)
This should address user reports of huge bankruptcy counts even when
they are relatively caught up. The root issue is that we sometimes
don't mark messages as read for some reason.
(imported from commit 8799305a8665f9ee239575e6e95f603f89c1d427)
This reverts commit 1147814b22fb9737a807057ddbdbe0e9554086e0.
This seems to with some probability screw up our Zephyr mirroring
script.
(imported from commit 4f82452f1b0ca98e6b895db020e071d2daa325e4)
This requires a puppet apply on each of staging and prod0 to update
the nginx configuration to support the new URL when it is deployed.
(imported from commit a35a71a563fd1daca0d3ea4ec6874c5719a8564f)
Client objects are immutable and there are very few of them, so caching them in
memory is cheap and saves a trip to memcached.
(imported from commit 300b9b402f4e509f86a7fd86b5f898dc3f43738f)
UserProfile.show_admin was intended to be a check for users that have
administrative rights in other realms, which we've harmlessly but
erroneously been using to check if they are an admin in their realm.
Use the more straightforward check instead, with a more intuitive
name.
(imported from commit d81050c7dbbb19e59c5e31750be303a4630e1456)
There will be browser errors on staging when this is deployed due to the socket
protocol changing.
(imported from commit f1eda5b5c2ec9c60c23b3ca96277a61debadf5bb)
I believe there may also be others. I'm still not sure why clients would be
sending open requests without session or csrf values in their cookies, though.
(imported from commit 7e9660c1c4d5c2abf55ff21b433ba0117180eb82)
Previously, we counted not just the time required to process a
particular request, but also the time required to import+find the view
function via urls.py. The latter is usually fast, but when a new
Django thread starts up, it can take significant time, resulting in us
flagging slow requests on each server restart and also when a new
Django thread starts up on prod to handle requests.
This change means that we no longer include that startup time as part
of request processing time -- but we still log it in the case that it
was more than 5ms, so that we can identify why a particular request
was slower than expected if high startup times become a problem. We
annotate the time in log lines as "+start" rather than just "start" to
make clear that it's not counted in the total.
(imported from commit c677682e23b26005060390d85d386234f4f463ad)
This is useful for the occasional case where we cannot figure out what
is causing a particular problem, but it can be easily reproduced on
staging.
(imported from commit 8b51184a8b686814f2c6ff103ba355538463ceb0)
CUSTOMER13 doesn't want it, and there's currently no nginx config
or configurable Camo URI, so it wouldn't work if image preview
were enabled.
(imported from commit 615d4a32acbc4d4d590f88cf4e7d45d8f49db1d3)
This fixes a problem where the desktop app would attempt to load
https://zulip.akam.ai:8888/ after authenticating the user, which fails
with CSS issues.
We should probably, separately, change our Django-under-apache to only
serve the one URL that it needs and redirect the rest back to
Django-under-nginx.
(imported from commit 3e3251863618269790f61b371e88af57b6cfb272)
The problem here is that some error-related templates render without
the context managers that we specify, and so we want the handling of
"the context was not set" to default to the enterprise case (because
the enterprise version will work basically fine on !enterprise, but
the opposite is not true because the enterprise distribution doesn't
even have the templates for certain !enterprise pages that may be
linked to).
(imported from commit 7547311d87e048d33221587f44b82fe0ba320ca1)
This command should be run continuously via supervisor. It periodically
checks for new email messages to send, and then sends them. This is for
sending email that you've queued via the Email table, instead of mandrill
(as is the case for our localserver/development deploys).
(imported from commit a2295e97b70a54ba99d145d79333ec76b050b291)
ScheduledJobs with type Email displace the usual mandrill codepaths
in the Zulip Enterprise deploys
* Email-specific helper functions will appear in deliver_email.py
* 0058_auto__add_scheduledjob.py
(imported from commit 8db08d8a279600322acfdbed792dc1a676f7a0ab)
In particular, EXTERNAL_HOST doesn't specify the protocol, which gets
coerced to HTTPS.
(imported from commit 53f2e8106cf33114dcdd2ad17e09b41609641e71)
Errors are sent to a queue processor that posts them to staging,
just like the feedback bot.
(imported from commit 4a8d099672a1b3e48a8bc94148d8b53db73d2c64)
Luke Faraone
Jessica McKellar
Waseem Daher
Tim Abbott
Jason Michalski
Zev Benjamin
Steve Howell
Leo Franchi
acrefoot
Kevin Mehall
Jeff Arnold
Allen Rabinovich
acrefoot