df4b2d9f85
Break out non-Twisted email mirror code into separate library
...
(imported from commit 45c3e70ba81cdb44f1e9db4f162e5f2d2fe5ead6)
2014-01-06 13:36:53 -05:00
266c0f4886
Remove duplicate validation in email_mirror
...
We already have a try-except earlier in the file about email_gateway_user, so we don't
need to check for it again.
(imported from commit 2d9fa357fab2605916c5c5cb61961c0a121b1211)
2014-01-06 13:36:53 -05:00
a20894d847
digest: fix generator bug not producing correct deployment comparison.
...
(imported from commit 33c34d71320fa1f8cd095a6d9f10b2397eeeda2f)
2013-12-31 16:34:52 -05:00
61103d8349
digest: On an app server, only send digest emails for domains it manages.
...
(imported from commit 9aea636a19eb6fdefacdc7dc393bf075a05862c5)
2013-12-20 12:50:23 -05:00
a8cd91d6c7
digest: send emails if you've been active for slightly less than a day.
...
That way if all you do is briefly check Zulip because you got the
email, we'll send you another one tomorrow.
(imported from commit fcbbd264c5e5fea7352f0fee6989e000af7b7bed)
2013-12-17 10:47:17 -05:00
6b64ef6be3
digest: log when we enqueue emails.
...
(imported from commit 1b34e05bc2eebedb818673a4c9a3944e1db15c6d)
2013-12-17 10:47:17 -05:00
71f5b4e0ae
Enable digest emails for zulip.com.
...
(imported from commit f676f96c2b516b33269b8f5c72deaf5c985e8ccd)
2013-12-17 10:47:17 -05:00
b8579c6848
Disconnect handlers if the client closes a connection.
...
Tested using the following procedure (run-dev.py won't pass through
the client connect closing)
tabbott@monastery:~/zulip$ curl http://localhost:9991/api/v1/register -u email:key -d 'event_types=["message"]'
{"msg":"","max_message_id":2259,"last_event_id":-1,"result":"success","queue_id":"1386884005:0"}
tabbott@monastery:~/zulip$ curl -G http://localhost:9993/api/v1/events -u email:key -d "last_event_id=0" -d "queue_id=1386884005:0"
(then hit ctrl-C)
(imported from commit 3c4f3d5caac97b3de53da994ff9cd9ef67b2b9ea)
2013-12-12 16:52:25 -05:00
4d30756e72
[manual] Add a manage.py command to import realm filters
...
This must be run manually on staging after deployment. Once it has been run,
it can be deleted. It only needs to be run on staging, not prod.
(imported from commit 79252c23ba8cda93500a18aa7b02575f406dd379)
2013-12-11 14:39:10 -05:00
e377be96d4
Add a management command to set realm filters.
...
(imported from commit 6590d7eb58ea6e6afc15104dd75ad56b832858fa)
2013-12-11 14:39:10 -05:00
1b5c1ac021
Update style of client strings.
...
(imported from commit 1516461cf53b2715de68e01f16bb8a8cc33c48ad)
2013-12-09 11:47:52 -05:00
e5be713103
Clean up EXTERNAL_API_HOST usage and defaults.
...
We now have 2 variablse:
EXTERNAL_API_PATH: e.g. staging.zulip.com/api
EXTERNAL_API_URI: e.g. https://staging.zulip.com/api
The former is primarily needed for certain integrations.
(imported from commit 3878b99a4d835c5fcc2a2c6001bc7eeeaf4c9363)
2013-12-04 15:10:54 -05:00
89844c6ced
Revise day1/day2 emails.
...
(imported from commit dc5e78cccb22b024be46b3066d44f09ea1b53345)
2013-12-02 11:15:13 -05:00
0c142015ea
Tool to test LDAP functionality
...
(imported from commit c4b4d8b25ba3afd333faf09bc232da2f92a57b7e)
2013-11-25 17:44:47 -05:00
4a8ec7b357
Management command to update names from the LDAP database
...
(imported from commit 1e1b12c055926899fdca3e484df6c9437c800c6c)
2013-11-25 17:44:47 -05:00
7cb261a0ed
Set default streams to engineering,social when creating realms.
...
(imported from commit 4e174fec162be4138da099edfc4cfb8d5662b7f1)
2013-11-25 14:00:28 -05:00
ca8225cf47
[manual] Add endpoint to cleanup a finished events queue.
...
This requires a puppet apply on each of staging and prod0 to update
the nginx configuration to support the new URL when it is deployed.
(imported from commit a35a71a563fd1daca0d3ea4ec6874c5719a8564f)
2013-11-20 18:34:15 -05:00
f3fa78d30c
Rename do_[rd]activate to do_[rd]activate_user.
...
For similarity with do_activate_user.
(imported from commit 2874af84dbe1ce666de10e0d8ffe2daa8bad9731)
2013-11-16 11:35:26 -05:00
9616ea7afd
remove enqueue_followup_emails management command
...
(imported from commit 20ce0b698cf795a6e1c7ff5ba7b546901e1fd8a4)
2013-11-15 21:31:37 -05:00
e77d534e78
remove ALLOW_ARBITRARY_SENDERS_LOCAL_EMAIL
...
(imported from commit 7a374959516d8505685466293f146d95094a7ea7)
2013-11-15 21:31:37 -05:00
c24addf2d5
Build internal bots in the zulip.com domain.
...
Otherwise do_create_realm can't actually send the notifications.
(imported from commit 4fa9a53f1d3d8a2e26d7b89401e7dfa77f2f7533)
2013-11-14 11:16:16 -05:00
e2537ade44
Create the admin realm using do_create_realm.
...
That way, we get the first-user message sent automatically.
(imported from commit 70be26a8152676055a94c363706c35964284a3de)
2013-11-14 11:16:16 -05:00
af2da58fd0
Set default default streams for enterprise installations.
...
(imported from commit 87b9fdc6ef19a88a2abda4ba5cdfe2a5b35d581d)
2013-11-13 21:42:21 -05:00
4078b8bb6c
Move set_default_streams from zilencer to zerver.
...
We'll want to use it for setting up local server instances.
(imported from commit 47c60c349ba3cebad58506f439013def4c40708b)
2013-11-13 21:08:14 -05:00
bf991dac94
enterprise: Create feedback_bot as a bot.
...
(imported from commit 8bd985f82aa80380188a644869c55bc54b6f7977)
2013-11-13 17:26:30 -05:00
6d38285a2e
fixup supervisor oops related to postmaster config
...
(imported from commit 8b5c39f0d13abb5e1def9f88a2ab82cfa67b42f6)
2013-11-13 17:15:55 -05:00
81156cf2f5
Clean up initialize_enterprise_db and fix domain.
...
(imported from commit af6f20c7705720c0043817516fdb3c21777796bb)
2013-11-13 16:47:37 -05:00
b134ecc102
add print_email_delivery_backlog for nagios reasons
...
(imported from commit c1c1edb8614913c7fff63e175423ee5701465223)
2013-11-13 16:41:36 -05:00
f5199081d5
Add the deliver_email management command
...
This command should be run continuously via supervisor. It periodically
checks for new email messages to send, and then sends them. This is for
sending email that you've queued via the Email table, instead of mandrill
(as is the case for our localserver/development deploys).
(imported from commit a2295e97b70a54ba99d145d79333ec76b050b291)
2013-11-13 16:41:36 -05:00
f5fc59e3e0
email mirror: remove some TODOs enterprise users don't need to see.
...
These should live in tickets anyway.
(imported from commit 9c2a9445640278d251a99c68c0074c138b832c60)
2013-11-13 16:28:30 -05:00
3b4303598a
email mirror: use the correct API host locally.
...
In particular, EXTERNAL_HOST doesn't specify the protocol, which gets
coerced to HTTPS.
(imported from commit 53f2e8106cf33114dcdd2ad17e09b41609641e71)
2013-11-13 16:28:29 -05:00
8382e074fe
Move use of html2text to a subprocess call.
...
(imported from commit 36e8a6f030d75196c28fbdc0e58c6968952d95ff)
2013-11-13 16:28:29 -05:00
691c861c85
Guard against creating a realm that is already an alias to another realm
...
(imported from commit 365c79148a1a3af190d2eab449d735525554657b)
2013-11-13 14:17:44 -05:00
62ba16a829
[schema] Allow a realm to have aliases for multiple other domains
...
(imported from commit 5d0340afd4c77f50db63550a3ec8d81e885c285c)
2013-11-13 14:17:43 -05:00
6bcd5a7ffc
Update installation instructions to include configuring the app.
...
(imported from commit 89395815c32416ec0636efcff12eb76f5d890bbd)
2013-11-13 12:02:50 -05:00
300cceb3e4
Create feedback bot in enterprise installs
...
(imported from commit 0c0a6919bd29e12484d520f80789b39b203099b0)
2013-11-13 11:18:03 -05:00
d7b823f017
Point email mirror to staging for zulip.com in non-localserver, and prod otherwise
...
(imported from commit 3076df1e68e0200af06b79a8cdeb9822bbf4c10c)
2013-11-12 22:22:13 -05:00
fafa14db9a
Patch out references to zilencer so the app runs without it on ENTERPRISE
...
(imported from commit bd4fe1ff47e1b4dc9038f800ee8aa2980bd00fdf)
2013-11-12 18:43:06 -05:00
3dadb5b76a
Update comments to rename Local Server => Enterprise.
...
(imported from commit 82bf89aa1148a78ce11ead203a0bc16243d1fca8)
2013-11-12 15:57:02 -05:00
21f1058fd2
Rename local_server=>enterprise in DB initialization script.
...
(imported from commit 76e26f43858e3baa6ed1b38d67973ed41acf04f3)
2013-11-12 15:57:02 -05:00
5293cdebe8
Rename LOCAL_SERVER to ENTERPRISE.
...
(imported from commit 7edf353eefe6c9e7aac74b7bbc37b923cac1b913)
2013-11-12 15:57:01 -05:00
a593a798f8
Move send_stats management command back to zerver/
...
It's not analytics, and it's needed for restart-server.
(imported from commit 979fa15715ea437cbbc5d986c859ee4d6c668da8)
2013-11-12 15:50:08 -05:00
64c3a9a0bd
initialize_local_server_db: Improve success output.
...
(imported from commit 7ac45876e639b7af5a283fb60b134ec13438a313)
2013-11-12 09:38:43 -05:00
96918e30f9
initialize_local_serverd_db: Create the realm for ADMIN_DOMAIN.
...
(imported from commit e13e73e54871c106cd4fd465b74c645657899dce)
2013-11-12 09:38:42 -05:00
e06722657a
[manual] Remove /messages/latest API and related legacy code.
...
This requires doing a puppet apply on our servers to take effect
properly.
(imported from commit 19dc56f071f07a5d2571eef49dd835121b2e82b6)
2013-11-05 14:19:40 -05:00
0bcc8db924
Give the initial bots themselves as owners
...
(imported from commit aca5565a3705aa6c6d55eb8afb5ca84f0efc9a0c)
2013-11-04 17:36:14 -05:00
97e9615921
Add script to replace populate_db for local server.
...
(imported from commit ca51391fc1d8ed13c0fcd36e17c875a085c78eb0)
2013-11-04 17:01:41 -05:00
fe1313ad76
Split management commands for internal use into zilencer/
...
(imported from commit 3fd81bfc7e987fc88bd1d632546850eebbb92234)
2013-11-04 11:58:52 -05:00
68dcc760c3
Clean up some unused imports.
...
(imported from commit 0c5d8e2a55ba1b8909ba807fee3afe863dcdc226)
2013-11-04 11:51:17 -05:00
7a67fb34b0
Rename: queue_followup_emails -> enqueue_followup_emails.
...
(imported from commit c44e790de5b694f8d98b03608231f7a284856254)
2013-11-04 10:20:59 -05:00
c4141eeeb0
Rename: queue_digest_emails -> enqueue_digest_emails.
...
(imported from commit a72fdf849814ff974117c55bf1b155c494a1b160)
2013-11-04 10:20:59 -05:00
03c5ae44a4
Remove unnecessary transaction in message flag updating.
...
We only needed a transaction here to workaround problems associated
with not having database-level autocommit.
(imported from commit 240ba05a4a4a846a7ff62e6e59e403ab0d78ab11)
2013-11-04 07:31:39 -05:00
54a25b065a
Add show_admins management command.
...
(imported from commit f16e118fa5203408b99f0a41ff7cecbffb821fa7)
2013-11-02 11:37:27 -04:00
ca8187b8cf
Email gateway bot variable name changed to EMAIL_GATEWAY_BOT
...
(imported from commit 01c430bd9b916c9bb152ee821214495333f1d37e)
2013-11-01 16:46:17 -04:00
db6550e99a
Make built-in bots come from settings.py, and allow localserver-specific ones
...
(imported from commit e21933e37487314ac986147562817a19227e8960)
2013-11-01 14:13:05 -04:00
ddbfe31449
Remove obsolete management command for mit.edufullname transition.
...
(imported from commit aed2a2d7bfedd98e2406bb18bf54f953a5ae7ef3)
2013-11-01 10:30:17 -04:00
11b92534e8
Remove obsolete management command for unread counts transition.
...
(imported from commit 52c6acbf4e8e53f86e880e475396d824c729405f)
2013-11-01 10:30:17 -04:00
429c41da79
Rename client-activity to client_activity to match other management commands.
...
(imported from commit b0b2cd372810923b3c808142f7b7eeef285d132d)
2013-10-31 15:29:50 -04:00
dbd4c187e3
Rename create-stream to create_stream to match other management commands.
...
(imported from commit 28daad12943ab5eb245d7737934b08f5b29e8f90)
2013-10-31 15:29:50 -04:00
5b1e504e13
Rename rename-stream to rename_stream to match other management commands.
...
(imported from commit 28522ac669a38e3de3f25b28c82ab0c787aecd0b)
2013-10-31 15:29:50 -04:00
013e84ef86
Remove obsolete explain_js_error management command.
...
(imported from commit 92f23e3bb7e6dbbad5162446b5f19e74728cf608)
2013-10-31 15:29:50 -04:00
217478e988
Remove obsolete clear_db management command.
...
do_destroy_rebuild_database is the "new" way to reset one's dev
database.
(imported from commit 8edb707ecd5e6e24b62eef324915cc3a4a5b8eda)
2013-10-31 15:29:50 -04:00
384f64d38d
Remove obsolete management commands from various field additions.
...
(imported from commit b44a2b683768c790839e82e2e9c9a18827670113)
2013-10-31 15:29:50 -04:00
f0580dd28c
Remove obsolete management command for activating MIT users.
...
(imported from commit a965eea8edfc9d73a00770d85cd517343f607338)
2013-10-31 15:29:50 -04:00
06d43a6f97
Remove obsolete management commands from the stream color transition.
...
(imported from commit d6d508d952714a1865ca4fddcb48d0ccedb6f4dd)
2013-10-31 15:29:50 -04:00
6d8fc371fb
Remove obsolete command to delete tutorial streams.
...
We don't have tutorial-bot anymore, so this is no longer needed.
(imported from commit b8a1bc9c2379a451e8cc1ed43b15f373bb3f2b06)
2013-10-31 15:29:50 -04:00
c8d07cc64c
Remove management commands from the mysql=>postgres migration.
...
(imported from commit 636f71b435a50e83fd8a9e40685d7c52b3b01f30)
2013-10-31 15:29:49 -04:00
1d784801cf
More settings.LOCALSERVER -> settings.LOCAL_SERVER.
...
(imported from commit 2b5045d302b8ebb2a5c7175264b4ab9cfef2844d)
2013-10-31 13:31:03 -04:00
70c1c31b3a
Simplify callback protocol for register_json_consumer().
...
The register_json_consumer() function now expects its callback
function to accept a single argument, which is the payload, as
none of the callbacks cared about channel, method, and properties.
This change breaks down as follows:
* A couple test stubs and subclasses were simplified.
* All the consume() and consume_wrapper() functions in
queue_processors.py were simplified.
* Two callbacks via runtornado.py were simplified. One
of the callbacks was socket.respond_send_message, which
had an additional caller, i.e. not register_json_consumer()
calling back to it, and the caller was simplified not
to pass None for the three removed arguments.
(imported from commit 792316e20be619458dd5036745233f37e6ffcf43)
2013-10-31 13:18:11 -04:00
ddc7608afe
Add enqueue_file management command.
...
This reads lines from a file and pushes them on to one of our
worker queues.
(imported from commit 2ed3338c5065819f8bafb247a15799e2bc9e7d39)
2013-10-30 16:47:16 -04:00
ab9b1be82c
Rework management commands to handle deployment assignment
...
We now ensure `create_realm` adds you to a default deployment and that
`create_deployment` removes the old deployment association when
performed.
(imported from commit 5b94fb07b8e11332765b057dc640a5ed873ec99e)
2013-10-30 12:00:13 -04:00
901a5c342c
Use os.path directly rather than sometimes importing it separately.
...
(imported from commit 48486c4ea64d02a15faeebb0f490d31e9b113d62)
2013-10-28 15:44:13 -04:00
807116e955
Delay connecting to RabbitMQ until it's necessary
...
Previously we were connecting at import-time.
(imported from commit 9a638f0d238f3b6b00feb4aa524098a64953cb92)
2013-10-28 15:27:25 -04:00
ffb6266319
Refuse to run a queue processor if USING_RABBITMQ is False
...
(imported from commit 39beff47cdbb18ba39756989e6f07facbd16864f)
2013-10-28 14:30:53 -04:00
8aa2c7b547
Make the purge_queue management command more efficient
...
Before we were removing items individually from the queue. We now
directly use RabbitMQ's queue purging mechanism.
(imported from commit 62ab52c724c5a221b4c81a967154a4046a579f84)
2013-10-28 14:30:53 -04:00
fac9d34d6c
digest: Add a management command to enqueue digest email recipients.
...
(imported from commit 70ff2c7a4dae654f4077041c45e2154b3ac7afb7)
2013-10-25 19:08:47 -04:00
66f48288b4
Rename LOCALSERVER -> LOCAL_SERVER.
...
(imported from commit b3abdd10d54d2ad7a9c463af9a291d2e2127707f)
2013-10-25 17:37:06 -04:00
a1b44986e0
[schema] Add a base_{api,site}_url property to our deployments.
...
This will allow us to redirect clients to the correct local site.
To apply this migration, just run:
python manage.py migrate zilencer 0002
(imported from commit 7bd39b5f035145b6b52e1b2cb2ad5f6720d598ce)
2013-10-25 15:18:37 -04:00
81d7dd1fda
[schema] Support for authenticating Deployments via the API.
...
Here we introduce a new Django app, zilencer. The intent is to not have
this app enabled on LOCALSERVER instances, and for it to grow to include
all the functionality we want to have in our central server that isn't
relevant for local deployments.
Currently we have to modify functions in zerver/* to match; in the
future, it would be cool to have the relevant shared code broken out
into a separate library.
This commit inclues both the migration to create the models as well as a
data migration that (for non-LOCALSERVER) creates a single default
Deployment for zulip.com.
To apply this migration to your system, run:
./manage.py migrate zilencer
(imported from commit 86d5497ac120e03fa7f298a9cc08b192d5939b43)
2013-10-25 14:13:30 -04:00
52309b5789
Enable absolute imports for non-third-party code where it was omitted previously
...
(imported from commit 34856ac32c6b94b614273a6fe46c87a314058d9b)
2013-10-25 14:13:30 -04:00
a9b9c0ec7b
Add Email Mirror default settings and error out with readable warning if not configured
...
(imported from commit adac321d09561b51277ec1e314a1d480dc01e0c8)
2013-10-25 11:51:13 -04:00
2d276179d0
Refactor email-mirror to handle running on any machine
...
(imported from commit 2971449ceaacb564770e66874fc095f77e68d445)
2013-10-25 11:37:34 -04:00
dc8bfde0b4
Add a check_apns_tokens manage.py command to flush old tokens
...
(imported from commit 15dcfaa1967bd749b582882c4a816a457ca6cc12)
2013-10-24 14:54:31 -04:00
05564ec0e2
[schema] Add a config option for enabling/disabling push notifications
...
(imported from commit 5649188cd5e0e4edb6e71559ec7d7fc5e92efaf8)
2013-10-24 14:54:30 -04:00
5979af3a45
[manual] Add asynchronous message sender via sockjs-tornado
...
New dependency: sockjs-tornado
One known limitation is that we don't clean up sessions for
non-websockets transports. This is a bug in Tornado so I'm going to
look at upgrading us to the latest version:
https://github.com/mrjoes/sockjs-tornado/issues/47
(imported from commit 31cdb7596dd5ee094ab006c31757db17dca8899b)
2013-10-22 18:45:11 -04:00
03c856ce10
Add create-stream management command.
...
The current version should only be used for testing; for example,
if you want to create a bunch of streams for stress testing, you
can run this in a loop.
(imported from commit ec51a431fb9679fc18379e4c6ecdba66bc75a395)
2013-10-19 09:28:59 -04:00
b7419d50b4
create_realm: add a reminder to run set_default_streams.
...
(imported from commit 59cf9378f8f271a88a6fbda591814fd9142a8d93)
2013-10-17 13:47:51 -04:00
6dbf7613a1
Supply a name when creating realms.
...
For our populate_db bulk creation, just use the domain.
(imported from commit 4fb756f6dfa2d8f90e55822e27891e84168d5d1c)
2013-10-17 13:47:51 -04:00
648b3bed74
create_realm: don't let us create realms with invalid domains.
...
(imported from commit 95a75a941b18d0857027084f5f9ea4c011805c9d)
2013-10-17 13:47:51 -04:00
753aace8f8
Add a basic statistics module to analyze active user engagement
...
(imported from commit 4d7a21b9a09b97c88a80d0446fb16e53012507c9)
2013-10-16 18:39:44 -04:00
98897fe5e3
Queue the day 1 and day 2 Zulip followup emails
...
Handled by the queue processor for signups. Added a management command
that accomplishes the same task, in case it's needed for manually added users,
or in case we goof and need to remove queued emails for a given user.
This addresses Trac #1807
(imported from commit 6727b82a07fa6a3ea3d827860c9e60fd0602297a)
2013-10-10 19:32:21 -04:00
6afdfeb1dc
Filter obvious footers from email mirror messages.
...
(imported from commit 7e22b6823b5d0a07f7f350e4af65be66728a8f88)
2013-10-02 14:39:22 -04:00
f0911f51a0
email mirror: ignore decoding errors.
...
Empirically, we only get these for malformed emails where the charset
specified in a message part header does not match the true encoding of
the part. I checked what the resulting Zulip looked like for the
original offender, and it looked find with ignoring errors.
(imported from commit ac6ba65b611cb22d4ec547b75a585abce6fc50b0)
2013-09-27 11:34:57 -04:00
e4496521e9
Update logging path for email-mirror.
...
(imported from commit 75f36dbb963b2b77ecda7ba138023f0f774047e0)
2013-09-25 16:52:41 -04:00
0d897c469d
Add a utility to report some rough recent client activity metrics.
...
(imported from commit 27b4a70871939b2728fcbe0ce5824326ff4decc2)
2013-09-25 14:32:53 -04:00
871afde142
Prepare process_queue to be used with supervisor's numprocs
...
(imported from commit 5a652b93f1e8b32b5ed89d622035161abaedfb11)
2013-09-24 20:44:15 -04:00
e450fc80c9
Move existing queue processing workers to new framework
...
(imported from commit b448eeefcddb1840344bd337046be30714a7fdbc)
2013-09-24 20:44:15 -04:00
6cf5ebc63d
Add new system for defining and running queue-processing workers
...
(imported from commit 7e77da57b8ad0b70837785c85f546601ba5b1957)
2013-09-24 20:44:15 -04:00
d434bf4bb4
email mirror: use a message part's specific charset rather than the message charset.
...
The overall message charset may be null or not match the part's
charset. Even though it's unclear from the documentation,
experimentally using the charset for a message part seems to give you
the charset even for non-multipart emails.
(imported from commit 0e1d23073f4c53041f9760e66a6635f8a94893d1)
2013-09-24 16:19:53 -04:00
7bb75e3acb
Populate date_created for existing realms.
...
(imported from commit 6a10150905a23589f098778ccbd8075025ff900f)
2013-09-24 16:19:52 -04:00
Luke Faraone
Jessica McKellar
Tim Abbott
Leo Franchi
Waseem Daher
acrefoot
acrefoot
Kevin Mehall
Steve Howell
Zev Benjamin
Jessica McKellar